Tag: ZeRO
-
Multi-GPU Fine-Tuning: DDP, ZeRO, FSDP and the Wire Between the Cards
Two GPUs solve two different problems. DDP for throughput, ZeRO and FSDP for memory, and why the interconnect between the cards decides…
Two GPUs solve two different problems. DDP for throughput, ZeRO and FSDP for memory, and why the interconnect between the cards decides…