A curated Megatron reading path: the parallelism map, what actually splits, how NVIDIA and TPU wrappers differ, and the migration surfaces around NAM56R-style layouts.
This hub is for readers who need the execution order of Megatron concepts, not just a glossary. Start with the parallelism map, then move into the boundary documents and finally the migration or recipe surfaces that turn those ideas into launchable layouts.
Best if you keep hitting TP, PP, EP, FSDP2, or Megatron wrapper terms across the blog and want one stable reading path.
Build the vocabulary and application order before touching wrappers or recipes.
What data, tensor, sequence, context, pipeline, and expert parallelism each own, how they compose, and where the real integration risks still live.
The shortest accurate map of the parallel axes MegaCpp actually uses.
A code- and doc-grounded walkthrough of tensor parallelism in public hybrid recipes, including where TP helps, where it does not, and how it fits into hybrid NAM52 and NAM56R workloads.
What really splits under tensor parallelism, what stays global, and why the difference matters later.
An explanation of SP versus CP using TP-aware helpers, long-context bring-up patterns, and hybrid model design.
The cleanest companion read once TP terminology starts colliding with long-context layout terms.
These explain where Megatron stops, where wrappers start, and how the layout lands on actual hardware.
A grounded look at split-friendly and split-hostile model surfaces: TP, SP, PP, EP, recurrent state, side embeddings, and why some boundaries remain architectural rather than automatic.
The boundary document for what the Megatron lane can express without custom seams.
How MegaCpp lays out the TP × PP × DP × EP cube on H200 multi-node systems and GB10, integrates DualPipe / DualPipeV with our hybrid layer pattern, accounts for pipeline bubbles, and launches the deployment training job.
How the parallel map behaves once the layout has to survive real H200 and GB10 execution.
How MegaCpp combines Megatron-Core DistributedDataParallel with PyTorch FSDP2 across H200 and GB10, the gradient-bucket sizing rules we ship, the freeze plan for the eight specialists, and the failure modes that defined the contract.
The NVIDIA-side wrapper and sharding story once Megatron is not the only layout owner anymore.
These finish the picture by showing how layouts are expressed and ported.
Why MegaCpp ports only what Megatron or Nemotron do not already provide, and why ambiguous mappings should fail closed instead of being reinterpreted silently.
The decision framework for staying native versus carrying narrow custom seams around Megatron.
Why translating NAM56R into Megatron-native syntax is a fail-closed planning step, not a blind string rewrite.
The concrete translation story for a MegaCpp hybrid model into Megatron-native terms.
Why MegaCpp keeps high-level recipe objects and then lowers them into a smaller native Megatron flag surface instead of treating one giant launcher as the source of truth.
The recipe-level companion piece when the conceptual map has to become real launch arguments.
Keep exploring
These hubs cover nearby parts of the blog without turning the archive into a giant taxonomy.
A curated GB10 and Blackwell reading path: consumer-versus-datacenter tensor paths, driver-visible false positives, arch-patch repros, and the serving or precision choices that survived contact with the hardware.
A curated Modal reading path: when the rented-GPU surface was useful, what broke on multi-GPU launches, how receipts were recorded, and how we kept the lane debuggable.
A curated Mamba3 reading path: why MegaCpp kept a hybrid stack, how the kernels evolved across CUDA, TileLang, and TPU, and where the runtime wins actually held.
A curated MLA reading path: the weight-absorption contract, Megatron-safe integration boundaries, dispatch and FP8 edges, and the adapter surfaces that keep MLA connected to the rest of the stack.
A curated evaluation reading path: verifier-first harnesses, ablation structure, benchmark receipts, and the evidence rules that keep comparisons from collapsing into anecdotes.
A curated TPU sparse-attention reading path: block-sparse contracts, Pallas kernel choices, SPMD sharding, and the runtime surfaces that keep long-context TPU work stable.
A curated path through the H200 lane: operator bring-up, step-time anatomy, memory pressure, and the NVIDIA kernel surfaces that actually moved the stack.
A curated reading order for TPU work: bring-up, PJRT and Torch/XLA boundaries, SPMD sharding, and the kernel/runtime traps that made TPU performance non-obvious.
A curated archive for the C++ data path: corpus selection, semantic enrichment, packaging into training artifacts, and the file-level durability choices that keep the pipeline sane.
A curated path through the expert stack: what the specialist path changed, how routing works, and how the parallelism map constrains the model layout.