A curated MLA reading path: the weight-absorption contract, Megatron-safe integration boundaries, dispatch and FP8 edges, and the adapter surfaces that keep MLA connected to the rest of the stack.
This hub is for readers who keep seeing MLA across H200, Megatron, and Blackwell posts and need one grounded path through the integration story. Start with the architecture and boundary documents, then move into dispatch or cache surfaces, and finish with the adapter and upstream follow-through.
Best if you want MLA as a real system boundary with concrete implementation tradeoffs, not just as a glossary term.
Read these first to understand what MLA changes and where the clean seams actually are.
Multi-Head Latent Attention in production: why DeepSeek's absorbed decode path is the right choice for KV cache, why it is the wrong choice for training, and how the C++ specialist ensemble uses both.
The core architectural readback for what MLA changes in projection layout, KV handling, and the weight-absorption contract.
How MegaCpp keeps MLA-specific compatibility logic behind a narrow adapter seam instead of scattering it through the whole builder path.
The best integration overview once the concept has to survive real Megatron ownership boundaries.
Why MegaCpp keeps MLA-specific normalization behind one shared adapter seam instead of leaking MLA conditionals through the whole attention builder stack.
The practical boundary document for where adapters, shared modules, and MLA responsibilities should stop leaking into each other.
Once the boundary is clear, these explain the parts of MLA that become hardware-shaped.
The NVIDIA side of Multi-Latent Attention in the MegaCpp ensemble: a fused down-norm-up projection, a fused split-RoPE-concat Triton kernel, a compressed KV cache, and how it all lands on Megatron-Core.
The NVIDIA-side implementation lane once MLA has to compete on real H200 and Blackwell hardware.
Why SparseMLA kernels that hardcode DeepSeek-sized dimensions fail to scale down cleanly to NAM56R-style shapes, and what a generalized contract changes.
A focused read on how the sparse MLA path generalizes dimensions without turning into one-off kernel glue.
Why SparseMLA needs an FP8-aware dispatch contract when Transformer Engine wrappers hide FP8 storage behind a bf16-looking logical surface.
The dispatch and precision edge where MLA starts colliding with FP8 policy and runtime safety.
Per-specialist KV cache layout, MLA cache after weight absorption, paged attention adoption status, and what changes between H200 and GB10 - including the MegaCpp serving plan.
A useful companion once MLA decisions start affecting cache residency and long-context serving surfaces.
These complete the story once MLA has to live inside the broader model stack.
The LoRA, QLoRA, DoRA, VeRA, and DyLoRA family behind MegaCpp specialists, the registry and lifecycle that turn adapters into versioned releases, the hot-swap runtime, and the inference-facing API they power.
The adapter-side companion for understanding how MLA interacts with LoRA-family seams and runtime metadata.
A focused walk-through of the Mamba-3, Sparse-MLA, Liger-Kernel and DSA upstream PRs we have prepared: the bug, the fix, and where each one currently sits.
What changed when the MLA lane had to move beyond the local tree and into upstream-facing patches.
Keep exploring
These hubs cover nearby parts of the blog without turning the archive into a giant taxonomy.
A curated GB10 and Blackwell reading path: consumer-versus-datacenter tensor paths, driver-visible false positives, arch-patch repros, and the serving or precision choices that survived contact with the hardware.
A curated Modal reading path: when the rented-GPU surface was useful, what broke on multi-GPU launches, how receipts were recorded, and how we kept the lane debuggable.
A curated Mamba3 reading path: why MegaCpp kept a hybrid stack, how the kernels evolved across CUDA, TileLang, and TPU, and where the runtime wins actually held.
A curated evaluation reading path: verifier-first harnesses, ablation structure, benchmark receipts, and the evidence rules that keep comparisons from collapsing into anecdotes.
A curated Megatron reading path: the parallelism map, what actually splits, how NVIDIA and TPU wrappers differ, and the migration surfaces around NAM56R-style layouts.
A curated TPU sparse-attention reading path: block-sparse contracts, Pallas kernel choices, SPMD sharding, and the runtime surfaces that keep long-context TPU work stable.
A curated path through the H200 lane: operator bring-up, step-time anatomy, memory pressure, and the NVIDIA kernel surfaces that actually moved the stack.
A curated reading order for TPU work: bring-up, PJRT and Torch/XLA boundaries, SPMD sharding, and the kernel/runtime traps that made TPU performance non-obvious.
A curated archive for the C++ data path: corpus selection, semantic enrichment, packaging into training artifacts, and the file-level durability choices that keep the pipeline sane.
A curated path through the expert stack: what the specialist path changed, how routing works, and how the parallelism map constrains the model layout.