This post is simply meant to share what I’ve noted down along the way. Consider it a humble personal notes edition: not a definitive guide, just an attempt to make sense of things.
Why a personal note-type poster?
The answer is simple: I learn more easily when I try to put my notes together in an organized form. If it helps someone else along the way, even better.
01 · Definitions
Computer systems that perform tasks autonomously: ingesting and analyzing enormous volumes of data, then recognizing patterns in that data.
Clusters of powerful processors that work in parallel to process multidimensional data sets and solve complex problems at extremely high speeds.
02 · Characteristics
Massive datasets. Parallelism as gradient sync per step. High throughput within scale-up and scale-out. FP16 to FP4, mid and low precision. RAG during inference.
Parallelism as task division. Tightly coupled workloads. Ultra low latency across the cluster. FP64 double precision. MPI (message passing interface). Extremely high read/write storage.
03 · Backend networks
Existing HPC exascale deployments favor Dragonfly and Dragonfly+ topologies, although fat-tree is emerging as clusters become hybrid-purpose. Dragonfly eliminates the network scaling problem by design: instead of building massive switch trees to handle intra and inter cluster communication, it uses a physical layout that maximizes local connections within groups and minimizes long-reach optics across groups, with adaptive routing software doing the heavy lifting.
On the AI side, fat-tree 2-tier architectures already win the race for backend network designs. Rail-optimized principles are fundamental for both single and multi plane deployments: flattened topologies minimize the hop count for GPU-to-GPU traffic while providing full bisection bandwidth across the fabric.
Oversubscription is where the two worlds differ most in practice. AI backend fabrics demand a 1:1 non-blocking ratio between GPU-facing ports and leaf-to-spine links. HPC backend networks can typically tolerate 2:1 or even 4:1. Hybrid clusters need a more balanced approach to stay lossless and low latency.
04 · Frontend networks and storage
Node-local SSDs. Training spends far more time processing data on GPUs than reading from storage, so data is staged onto fast local NVMe drives ahead of time. The frontend network is separate from the backend, with shared bandwidth to remote storage, and a cost-effective design is preferred.
Shared parallel file systems. Workloads sync massive simulation checkpoints and read global datasets continuously, so any node must reach the centralized storage cluster at peak speed. Uniform ultra-high-bandwidth fabrics and dedicated, performance-optimized storage fabrics are preferred.
05 · Job scheduling
Both worlds tend to land on Slurm: it fits the bulk-synchronous, batch-style nature of large-scale LLM training on the AI side, and the queue management of large-scale parallel simulations on the HPC side.
06 · Traffic patterns across compute paradigms
- Bulk-synchronous on/off: a compute-idle fabric alternates with full-rate collective bursts.
- Gradient synchronization: AllReduce / AllGather over RDMA (NCCL / RCCL).
- Elephant flows with very high peaks.
- Network impact: synchronized incast, microburst congestion, tail-latency bound (the slowest flow gates the step).
- Irregular periodic bursts; many concurrent independent jobs on a shared fabric.
- MPI collectives at app cadence: CUDA-aware MPI (inter-GPU) plus parallel file system I/O.
- Many medium flows; small control and bulk halo messages coexist.
- Network impact: spine incast, head-of-line blocking, MPI and I/O link sharing.
- CCL training, MPI simulation and I/O sharing one fabric; multiple uncorrelated periodicities superimposed, so the fabric is rarely idle.
- Bimodal flow sizes: elephant (CCL, checkpoints) plus medium (MPI halos), with a volatile frame-size mix.
- Network impact: cross-job phase collisions compound incast and inter-tenant interference.
07 · References
Studies
- Understanding Data Movement Patterns in HPC: A NERSC Case Study
- A Study of Network Congestion in Two Supercomputing High-Speed Interconnects
- Characterizing the Impact of Congestion in Modern HPC Interconnects
- Modeling and Analysis of Application Interference on Dragonfly+
Definitions
Publications
- Resilient AI Supercomputer Networking using MRC and SRv6
- Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures
- Introducing Virgo Network, Google’s scale-out AI data center fabric
Prefer the original layout? The poster is also available as a 3-page PDF.
download the poster [ -> ]
Leave a Reply