Discover · feedburner.com

SE Radio 747: Sabina Anja on How AI Workloads Shape GPU Network Design

JUST™ Creative feedburner.com

Sabina Anja, Chief Technologist and executive advisor at VMware Cloud Foundation, talks with host Amey Ambade about how AI workloads reshape GPU network design, and why the network moves into the critical path of computation once a model is too large for a single GPU. Using Llama 3's 405 billion parameters as a starting point, Sabina explains how sharding spreads model weights across many GPUs. She then walks through the difference between scaling up inside a server over NVLink and scaling out across servers over the cluster network. She describes collective traffic as synchronous: no GPU moves to the next step until every GPU finishes, so one slow GPU or congested link, called a straggler,…

The discussion then turns to congestion and latency: why synchronized collectives produce sudden bursts that overflow switch buffers, how older techniques such as priority flow control return to pause traffic before loss, and how overlapping communication with computation hides some network delay. On inference, Sabina explains how sharded models pay network latency on every generated token, why the prefill and decode phases have different resource demands, and why some systems separate them into different GPU pools and move the KV cache between them. She also describes how mixture-of-experts models can concentrate traffic on the servers that host popular experts. The episode closes with a…

Length
53:46
More ways to use this
Open on another device
Scan this QR code to open the same public page

Scan to open this page. Your saved notes and account access are not included.

Download QR code

Open the page link

Names and references

  1. Network — person
  2. Design — person

Keep going

Each of these is read from what this estate already kept: the feed a piece came through, the publisher behind it, the credits and topics the publisher printed. None of it is a recommendation, an identity or a record kept anywhere else.

Also from JUST™ Creative

  1. SE Radio 746: Marian Kamenistak on Product Management for Software EngineersSE Radio feedburner.com
  2. SE Radio 745: Andrew Tridgell on Flight Control SoftwareSE Radio feedburner.com
  3. SE Radio 744: Danilo Krummrich on Rust in the Linux KernelSE Radio feedburner.com
  4. SE Radio 743: Sowjanya Pandruju on Managing Agent StateSE Radio feedburner.com
  5. SE Radio 742: Tim Varelmann on Mathematical Optimization for Software EngineersSE Radio feedburner.com
  6. SE Radio 741: Sahil Walia on Schema Changes and Data MigrationsSE Radio feedburner.com