Training on 8x H200 SXM: the operator playbook

An 8x H200Quick term guideH200NVIDIA's Hopper H200 GPU platform, typically discussed here as an 8-GPU training node with large HBM capacity and NVLink-connected ranks.GroundingH200 memory geometry training speed anatomy on H200 SXM node is a practical unit for training a mid-sized specialist model from scratch. On paper it looks like a larger-memory Hopper system. In practice the gap between a fresh machine and steady-state high-throughput training with reliable checkpoints is a sequence of small operational choices that, taken in the wrong order, cost days. This post focuses on that operator surface: how to drive the node, what the topology forces on the launch flow, which NCCLQuick term guideNCCLNVIDIA's collective-communication library for all-reduce, all-gather, reduce-scatter, and point-to-point transport on CUDA multi-GPU lanes.GroundingAbout: NCCL and collective hangs Example: pipeline parallel sample Reference: runtime optimization receipts settings are worth making explicit, and how we keep receipts comparable. First touch: NCCL hang triage means separating communicator bootstrap failures, compile-era watchdog timeouts, and steady-state collective skew before you change launcher policy, and a receipt is the compact run record that carries topology, env overlay, effective lane, and the first bounded step window together. For the knobs that become active once the launch lands, the closest follow-ups are H200 memory geometry, gradient accumulation and microbatching, checkpoint format and resume, and NCCL and collective hangs.

Why the operator surface is the contract

Training on this class of hardware is not about one heroic run. It is about repeating comparable launches across the same hardware class. That is a discipline problem, not just a performance problem. The launch surface is the contract. If two operators can produce different steady-state throughput on the same configuration because one forgot to pin a compile cache or left a stale NCCLQuick term guideNCCLNVIDIA's collective-communication library for all-reduce, all-gather, reduce-scatter, and point-to-point transport on CUDA multi-GPU lanes.GroundingAbout: NCCL and collective hangs Example: pipeline parallel sample Reference: runtime optimization receipts setting behind, the comparison stops meaning anything.

The H200 memory geometry changes the shape of the decision surface, but it does not eliminate activation pressure. A model that fits comfortably at short context can still run into trouble once sequence length, activation checkpointing policy, or expert routing buffers move together. We want one launcher that lands cleanly in both regimes and escalates predictably when it does not.

The practical framing versus H100 is simple: the common eight-way H100 SXM lane gives you 80 GB per GPU, while the matching H200Quick term guideH200NVIDIA's Hopper H200 GPU platform, typically discussed here as an 8-GPU training node with large HBM capacity and NVLink-connected ranks.GroundingH200 memory geometry training speed anatomy on H200 SXM lane gives you 141 GB per GPU. That bigger H200Quick term guideH200NVIDIA's Hopper H200 GPU platform, typically discussed here as an 8-GPU training node with large HBM capacity and NVLink-connected ranks.GroundingH200 memory geometry training speed anatomy on H200 envelope often lets you postpone the most aggressive checkpointing policy at the same batch shape, but it does not remove the need for one once context length, routed-token buffers, or activation fan-out start rising together. The adjacent decision surfaces are H200 memory geometry, activation checkpointing deep dive, and activation checkpointing policy.

Topology, launcher, and what the wrapper owns

Each rank owns one H200Quick term guideH200NVIDIA's Hopper H200 GPU platform, typically discussed here as an 8-GPU training node with large HBM capacity and NVLink-connected ranks.GroundingH200 memory geometry training speed anatomy on H200. The eight GPUs sit on an NVLink and NVSwitch fabric, and NVIDIA's public H200Quick term guideH200NVIDIA's Hopper H200 GPU platform, typically discussed here as an 8-GPU training node with large HBM capacity and NVLink-connected ranks.GroundingH200 memory geometry training speed anatomy on H200 materials frame that fabric as a first-class part of the Hopper-era multi-GPU story. What the operator has to get right is that NCCLQuick term guideNCCLNVIDIA's collective-communication library for all-reduce, all-gather, reduce-scatter, and point-to-point transport on CUDA multi-GPU lanes.GroundingAbout: NCCL and collective hangs Example: pipeline parallel sample Reference: runtime optimization receipts actually uses the expected topology and that nothing in the host image is silently routing traffic through a slower path, which is the same failure surface that later shows up in NCCL and collective hangs.

A fresh node still deserves one hard preflight: nvidia-smi topo -m should show the expected NVLink or NVSwitch fabric rather than an accidental fallback path.

On a healthy eight-way node, the GPU-to-GPU cells should read NV18 across the matrix. If you see SYS or PHB, stop and fix the topology first: that is a fallback onto CPU or PCIe paths, and any later NCCLQuick term guideNCCLNVIDIA's collective-communication library for all-reduce, all-gather, reduce-scatter, and point-to-point transport on CUDA multi-GPU lanes.GroundingAbout: NCCL and collective hangs Example: pipeline parallel sample Reference: runtime optimization receipts tuning result will be noise. The same preflight is a good time to check that host automatic NUMA balancing is off and to run a quick all_reduce_perf sanity pass before attaching the full training stack.

One knob is worth singling out because older recipes get it backwards: CUDA_DEVICE_MAX_CONNECTIONS=1 is not a safe H200Quick term guideH200NVIDIA's Hopper H200 GPU platform, typically discussed here as an 8-GPU training node with large HBM capacity and NVLink-connected ranks.GroundingH200 memory geometry training speed anatomy on H200/FSDP2Quick term guideFSDP2PyTorch's Fully Sharded Data Parallel v2 wrapper API. On CUDA it shards parameters, gradients, and optimizer state across the data-parallel group; in the TPU/XLA posts here it is usually a memory-goal analogy, not the actual eager wrapper mechanism.GroundingAbout: FSDP2 on XLA TPU History: FSDP2 pain and payoff Example: FSDP sharding sample baseline. On this lane it tends to serialize the very all_gather and reduce_scatter overlap that keeps step time sane. Unless you have measured reason to do otherwise, leave it unset and record that fact in the receipt.

The broader lesson is more stable than any single knob: environment-sensitive runtime behavior belongs in the receipt surface, not in tribal memory. That is the same reporting contract we use in Profiler and performance reports. The checked-in compile runtime env sample, compile/runtime receipt sample, and GPU profile receipt sample are the compact local versions of that environment-and-provenance contract.

One detail from the longer H200Quick term guideH200NVIDIA's Hopper H200 GPU platform, typically discussed here as an 8-GPU training node with large HBM capacity and NVLink-connected ranks.GroundingH200 memory geometry training speed anatomy on H200 memory story is worth keeping visible here: expandable_segments:True is allocator-fragmentation control, not a magic model-size increase. On long runs with changing routing or batch shape, the failure often shows up as reserved drifting away from allocated even though the model math did not change. The more detailed readback is in why a 4B-8B model fills an H200 and still OOMs; this post keeps it in the operator checklist because late allocator drift is operationally indistinguishable from "the node just became unreliable" unless the receipt names it.

PyTorch's current CUDAQuick term guideCUDANVIDIA's GPU programming stack: compiler, runtime, driver, libraries, and kernel toolchain used by CUDA training and inference lanes.GroundingAbout: XLA vs CUDA stack decisions History: GB10 tensor-path proof summary memory notes name PYTORCH_ALLOC_CONF as the allocator control and keep PYTORCH_CUDA_ALLOC_CONF as a backward-compatible alias. If an existing launcher still exports the alias, the launch invariant is that the receipt records which allocator overlay actually reached the CUDAQuick term guideCUDANVIDIA's GPU programming stack: compiler, runtime, driver, libraries, and kernel toolchain used by CUDA training and inference lanes.GroundingAbout: XLA vs CUDA stack decisions History: GB10 tensor-path proof summary context.

What the wrapper owns

A good launcher owns provenance, environment, the torchrun line, and log extraction. It should record the source revision, a short working-tree status, and the machine class into the log before the first Python import. That provenance boundary is the operator-side companion to Checkpoint format and resume.

It also has to preserve the node's real topology. The safe default is one node-local process group over all eight GPUs, with the receipt recording the effective mapping, rather than a wrapper that permutes local device IDs and leaves NCCLQuick term guideNCCLNVIDIA's collective-communication library for all-reduce, all-gather, reduce-scatter, and point-to-point transport on CUDA multi-GPU lanes.GroundingAbout: NCCL and collective hangs Example: pipeline parallel sample Reference: runtime optimization receipts reasoning over a partial or renumbered view of the host. If the launcher hides the topology, the preflight and the later throughput receipt stop describing the same lane.

That is also why MegaCpp prefers torchrun or srun-style launchers over per-process GPU masking schemes. If each worker only sees a renumbered CUDA_VISIBLE_DEVICES slice and believes it owns its own cuda:0, topology discovery becomes less trustworthy, NVLink assumptions get harder to reason about, and later NCCLQuick term guideNCCLNVIDIA's collective-communication library for all-reduce, all-gather, reduce-scatter, and point-to-point transport on CUDA multi-GPU lanes.GroundingAbout: NCCL and collective hangs Example: pipeline parallel sample Reference: runtime optimization receipts symptoms stop matching the preflight you ran on the full node.

That launcher choice matters for the network side too. On H200Quick term guideH200NVIDIA's Hopper H200 GPU platform, typically discussed here as an 8-GPU training node with large HBM capacity and NVLink-connected ranks.GroundingH200 memory geometry training speed anatomy on H200-class NVSwitch nodes, hiding the full local GPU set behind per-process masking can leave the full-node nvidia-smi topo -m preflight and the later NCCLQuick term guideNCCLNVIDIA's collective-communication library for all-reduce, all-gather, reduce-scatter, and point-to-point transport on CUDA multi-GPU lanes.GroundingAbout: NCCL and collective hangs Example: pipeline parallel sample Reference: runtime optimization receipts communicator describing different local maps. NVIDIA's NCCLQuick term guideNCCLNVIDIA's collective-communication library for all-reduce, all-gather, reduce-scatter, and point-to-point transport on CUDA multi-GPU lanes.GroundingAbout: NCCL and collective hangs Example: pipeline parallel sample Reference: runtime optimization receipts tuning guidance says the library selects algorithms from detected topology, including NVLink domain size, and keeps NVLS enabled by default on supported NVLink systems. That is why the receipt should keep the effective local rank mapping and NIC affinity beside the launch line: a lane that looked like clean NV18 fabric in preflight still needs a matching communicator view once collectives start. NCCL and collective hangs plus training speed anatomy on H200 are the right next surfaces when a supposedly healthy node suddenly loses overlap.

That same rule gets stricter once the lane crosses a single node or starts using RDMA-attached storage and networking. NV18 across the local GPU fabric is necessary, but it is not sufficient: you also want the receipt to keep GPU-to-NIC affinity and the actual communicator map visible so that a multi-node slowdown does not get misread as an intra-node NVLink regression. For the operator-facing follow-up, NCCL and collective hangs covers the failure taxonomy and comms cost and overlap covers the "healthy but slower" case where the node topology is fine and the overlap budget is not.

exec torchrun --standalone --nproc_per_node=8   -m <training-entrypoint>   --config "$CONFIG" --run_name "$RUN_NAME"   --device_batch_size "$DBS" --max_seq_len "$SEQ"

State on disk and live monitoring

There are three categories of state and they should go to three places.

• persistent training state belongs on a durable high-capacity data volume
• per-process artifacts such as compile caches belong in per-run scratch space
• operator receipts belong with the launch materials for that run

In practice the split is about traffic shape, not naming purity. Training reads want sustained throughput from the intended data root, checkpoint writes want burst-safe durable storage, and receipts should stay small and easy to collect beside the launch materials instead of disappearing into scratch. If those three paths collapse into one blurry directory tree, input stalls, checkpoint pauses, and provenance loss all look like the same "slow node" failure.

The invariant that should not drift is simple: the training data root must point at the intended data volume, not at an incidental home-directory path. Resume semantics only stay trustworthy if that split matches Checkpoint format and resume.

The useful operational addition is that the split is not just durable versus temporary. It is read bandwidth versus burst-write safety versus metadata noise. Keep the dataloader on the path built for sustained reads, keep checkpoint bursts on storage that can absorb them without stalling the node, and keep compile caches off that same hot path so a flood of small files does not masquerade as a training regression.

Takeaway

The H200Quick term guideH200NVIDIA's Hopper H200 GPU platform, typically discussed here as an 8-GPU training node with large HBM capacity and NVLink-connected ranks.GroundingH200 memory geometry training speed anatomy on H200 operator story is not that one knob solved training. It is that topology, launcher policy, and receipt discipline together make the node trustworthy enough to compare runs.

Once the node is stable, the next knobs are memory and batch-shape policies rather than topology. H200 memory geometry, Gradient Accumulation and Microbatching Under FSDP2, Training speed anatomy on H200, and CPU Offload and Startup Memory Calibration on H200 and GB10 are the next operator-facing layers of the same workflow. If you want the broader bring-up companion rather than one more knob-specific article, continue to H200 bring-up and naming.