A curated Mamba3 reading path: why MegaCpp kept a hybrid stack, how the kernels evolved across CUDA, TileLang, and TPU, and where the runtime wins actually held.
This hub works best in sequence. Start with the hybrid-model story and the author-level spec, then move into the kernel lane and finally the runtime or cache surfaces that make the Mamba3 path concrete.
Best if you want the Mamba3 lane as one connected engineering story instead of scattered kernel notes.
Read these first to understand why Mamba3 stayed in the stack.
A grounded look at why MegaCpp combines Mamba-style state-space blocks with a smaller number of attention blocks for long-context C++ work, and which parts are design choice versus published literature.
The top-level explanation of why MegaCpp kept a Mamba3-plus-transformer hybrid for C++ workloads.
A code-grounded explanation of how interleaved schedules work for NAM52 and NAM56R-style hybrid models, based on hybrid pattern notes, scheduling examples, and authoritative parallelism references.
The execution-plan companion that explains why A/M/E/R schedules need typed layer roles instead of vague hybrid branding.
Why an author-pure Mamba3 path still needs an explicit pre-projection RMSNorm when it is wrapped into a Megatron-local Mamba stack.
The author-side spec readback once the model had to fit Megatron and the wider stack honestly.
Why MegaCpp refuses to silently remap unsupported hybrid block families when translating NAM56R-style patterns into Megatron-native plans.
The translation boundary for unsupported hybrid block families once the Mamba3 lane has to survive a Megatron-native plan.
MIMO scaling, chunk-size behavior, the PsiV cache trade-off, and an honest tally of where a Mamba 3 hybrid outran pure attention on NVIDIA H200 and where it did not.
Where the Mamba3 path beat attention, where it lost, and why the answer changes with parallel layout.
These are the concrete implementation notes once the architecture story is clear.
How the Mamba 3 kernel stack works in MegaCpp: TileLang on H200, Pallas on TPU v6e, a CuTe DSL port that was evaluated but not adopted, and what each attempt showed.
The end-to-end kernel story across CUDA, Pallas, TileLang, and CuTe DSL tradeoffs.
Why some Mamba3-style kernels need an explicit 3D-to-2D shared-memory legality rewrite before the backend will accept the tile layout.
A focused deep dive into one of the more hardware-shaped Mamba3 kernel surfaces.
How we took the Mamba-3 trapezoidal SSM update from a CUDA Triton kernel to a Pallas/XLA-friendly scan on TPU v6e, and what survived the deployment port.
The TPU-side kernel note once the Mamba3 lane had to survive a very different backend.
These finish the picture with runtime and adjacent architecture surfaces.
Why the Mamba3 PsiV cache path is published as a scaffold with a fail-closed gate instead of a silent fallback.
The cache scaffold and runtime shape decisions that matter once the kernels themselves are in place.
Where matrix-state RNN layers, causal n-gram Engram branches, and the learned concept bank fit inside our Mamba 3 + Transformer hybrid — and which pieces remain useful in the public memory stack.
A useful adjacent read when the Mamba3 story turns into broader memory and recurrence design.
A focused walk-through of the Mamba-3, Sparse-MLA, Liger-Kernel and DSA upstream PRs we have prepared: the bug, the fix, and where each one currently sits.
The upstream follow-through when the Mamba3 lane had to be pushed beyond the local tree.
Keep exploring
These hubs cover nearby parts of the blog without turning the archive into a giant taxonomy.
A curated GB10 and Blackwell reading path: consumer-versus-datacenter tensor paths, driver-visible false positives, arch-patch repros, and the serving or precision choices that survived contact with the hardware.
A curated Modal reading path: when the rented-GPU surface was useful, what broke on multi-GPU launches, how receipts were recorded, and how we kept the lane debuggable.
A curated MLA reading path: the weight-absorption contract, Megatron-safe integration boundaries, dispatch and FP8 edges, and the adapter surfaces that keep MLA connected to the rest of the stack.
A curated evaluation reading path: verifier-first harnesses, ablation structure, benchmark receipts, and the evidence rules that keep comparisons from collapsing into anecdotes.
A curated Megatron reading path: the parallelism map, what actually splits, how NVIDIA and TPU wrappers differ, and the migration surfaces around NAM56R-style layouts.
A curated TPU sparse-attention reading path: block-sparse contracts, Pallas kernel choices, SPMD sharding, and the runtime surfaces that keep long-context TPU work stable.
A curated path through the H200 lane: operator bring-up, step-time anatomy, memory pressure, and the NVIDIA kernel surfaces that actually moved the stack.
A curated reading order for TPU work: bring-up, PJRT and Torch/XLA boundaries, SPMD sharding, and the kernel/runtime traps that made TPU performance non-obvious.
A curated archive for the C++ data path: corpus selection, semantic enrichment, packaging into training artifacts, and the file-level durability choices that keep the pipeline sane.
A curated path through the expert stack: what the specialist path changed, how routing works, and how the parallelism map constrains the model layout.