One morning of bugs

A training wave is a good stress test of every dependency in the stack at once. On one representative morning in April 2026 our hybrid Mamba-3 plus DSAQuick term guideDSADeepSeek Sparse Attention: a sparse-attention lane where routing and masking logic must stay faithful to the score path without breaking runtime constraints such as CUDA graph capture.GroundingAbout: DSA and CUDA graph safety History: DSA index cache patch Example: DSA CUDA graph safety sample recipe broke in four separate upstream libraries before lunch, and by the afternoon we had four draft upstream fixes staged in our patch lane. This is what that morning looked like, in the order the bugs appeared, and why we now treat upstream breakage as a planning constant rather than an exception.

Why MegaCpp cares about this

MegaCpp sits on top of a stack under active development: nightly PyTorch, current accelerator libraries, current Megatron-CoreQuick term guideMegatron CoreThe NVIDIA framework surface MegaCpp ports into through narrow adapters, layer specs, and runtime ownership bridges.GroundingAbout: Porting to Megatron friction About: Nemotron-style recipe as pure Megatron CLI Example: Mamba3 TP mixer sample, current mamba_ssm, TileLangQuick term guideTileLangA CUDA kernel DSL/compiler surface used here for explicit tile layouts, shared-memory legality fixes, and TMA-oriented kernel experiments.GroundingAbout: TileLang TMA and H200 reality History: upstream PR: TileLang and Megatron Example: TileLang TMA bulk-copy sample near the tip of development, Liger-Kernel, and Triton nightly builds. Pretending that dependency graph is stable is how you lose a training window to a regression that nobody outside your run would have noticed.

The product consequence matters even more. Production MegaCpp inherits whatever we choose to work around in research. If we only patch the symptom and never write the upstream-quality fix, we carry that patch indefinitely. Waiting passively for upstream is not viable on a fast-moving stack. The middle path is a patch lane: fix locally when needed, prepare the upstream-quality explanation quickly, and retire the local patch when upstream absorbs it.

What we built in practice

First kernel call of the day: DSA under CUDA graphs

The first symptom was the training launcher crashing before the first optimizer step. The recipe had enabled CUDA graphsQuick term guideCUDA GraphsCUDA's capture-and-replay execution model, where hidden host sync points or Python-side branching break an otherwise valid GPU work graph.GroundingAbout: DSA and CUDA graph safety Example: DSA CUDA graph safety sample Example: CUDA graph block validation sample through Transformer EngineQuick term guideTransformer EngineNVIDIA's Transformer Engine library path for accelerated Transformer modules and lower-precision training surfaces such as FP8, kept behind optional adapter seams in these posts.GroundingAbout: Transformer Engine on H200 and Blackwell-class GPUs: the bridge we use Reference: NVIDIA Transformer Engine documentation Reference: Transformer Engine FP8 and FP4 primer, and the indexer inside Megatron's DSAQuick term guideDSADeepSeek Sparse Attention: a sparse-attention lane where routing and masking logic must stay faithful to the score path without breaking runtime constraints such as CUDA graph capture.GroundingAbout: DSA and CUDA graph safety History: DSA index cache patch Example: DSA CUDA graph safety sample attentionQuick term guideAttentionThe token-mixing path that turns Q/K/V style projections into context-aware activations. On MLA pages here it refers to the concrete attention module boundary, not the A/M/E/R block-family shorthand.GroundingAbout: fused MLA on NVIDIA Reference: shared MLA adapter boundaries Reference: public-safe MLA integration patterns module failed during graph captureQuick term guideCUDA GraphsCUDA's capture-and-replay execution model, where hidden host sync points or Python-side branching break an otherwise valid GPU work graph.GroundingAbout: DSA and CUDA graph safety Example: DSA CUDA graph safety sample Example: CUDA graph block validation sample with cudaErrorStreamCaptureUnsupported.

Reading the source showed why. The hot forward path still contained CPU-synchronizing checks: several torch.equal(...) assertions and a branch around sentinel handling in _scatter_topk_into_index_mask. The reader-facing concept is the index_maskQuick term guideindex maskThe mask tensor DSA updates after sparse top-k selection so only the chosen key positions remain available to the later score path.GroundingAbout: DSA index cache patch History: DSA and CUDA graph safety Example: DSA CUDA graph safety sample update: scatter the selected top-k key slots back to legal values while sentinel or blocked positions stay masked. Those checks are harmless in eager mode and illegal during capture because they pull values back through .item(). The public-safe reproducer trail for that failure is DSA CUDA graph safety sample, DSA CUDA graph safety nearcopy, and CUDA graph block validation sample.

The fix was mechanical once we saw it. The eager-only assertions were gated away during capture, and the sentinel handling was rewritten into a branchless clamp-scatter-fixup. The lesson was not just "CUDA graphsQuick term guideCUDA GraphsCUDA's capture-and-replay execution model, where hidden host sync points or Python-side branching break an otherwise valid GPU work graph.GroundingAbout: DSA and CUDA graph safety Example: DSA CUDA graph safety sample Example: CUDA graph block validation sample are picky." It was that eager-mode safety checks can silently become graph-capture bugs if nobody audits them as execution modes evolve.

Second failure, same module, different shape

Ten minutes later, a different shape of the same run died with a different symptom: a large fp32 intermediate consumed an unreasonable amount of HBM before the DSAQuick term guideDSADeepSeek Sparse Attention: a sparse-attention lane where routing and masking logic must stay faithful to the score path without breaking runtime constraints such as CUDA graph capture.GroundingAbout: DSA and CUDA graph safety History: DSA index cache patch Example: DSA CUDA graph safety sample indexer even reduced over heads.

The culprit was an implementation that materialized the full score tensor and only then reduced it. At our shapes, that intermediate was allocated, consumed once, and discarded. The run ceiling was not determined by the full step; it was determined by one transient tensor that ate all the slack.

The fix was a straightforward reduction rewrite: stream per head, accumulate directly into the smaller destination tensor, and never materialize the full four-axis temporary. The broader lesson is that memory bugs are often hiding inside mathematically correct code that simply chose the wrong intermediate.

Third: fused linear cross-entropy with reduction="none"

The hybrid head uses a fused linear cross-entropy path with reduction="none" so we can apply a loss maskQuick term guideloss_maskThe per-token training mask that decides which positions contribute to loss after packing, FIM rearrangement, or documentation-aware masking.GroundingAbout: document masking and curriculum Example: packed rows schema sample Example: FIM long-context metadata sample per token before reducing. The forward looked fine. The backward produced grad_norm = NaN almost immediately, and the next iteration crashed with an illegal memory access.

Stepping through the implementation made the failure mode clear. The backward path multiplied stored gradients by grad_output using a kernel that only handled scalar-style broadcasting. That is valid when grad_output is truly scalar or effectively uniform. It is wrong when grad_output carries non-uniform per-token weighting, which is exactly what a masked loss does.

This is the kind of bug that can hide for a long time because simple tests do not exercise the real reduction shape. The fix was to route the local training path through a mathematically correct reduction strategy and write the upstream explanation around the actual masked-loss use case instead of a toy example.

Fourth: Mamba-3 MIMO with intermediate GQA grouping

The last bug of the morning came from a new preset using an intermediate GQA grouping on the Mamba-3 MIMO path. The backward path raised a literal unsupported-value error because the implementation only handled the two extremes: fully shared grouping and per-head grouping. The middle case had simply never been wired in.

The fix was to add the missing grouped branch and verify that it converged to the existing implementations at the edge cases. That same reproducer also surfaced a separate dtype issue in the wrapper stack: a module-level mixed-precision wrapper was silently casting parameters that the kernel path expected to stay in fp32.

That pair of bugs is a good example of why narrow reproducers matter. One feature addition exposed both a missing math branch and an integration-layer dtype assumption.

By mid-morning we had multiple live upstream bugs, one corner case that crossed library boundaries, and a patch lane with several new entries.

How it lands in MegaCpp

In production MegaCpp, each of these becomes a tracked patch-lane entry with a clear retirement condition.

The DSAQuick term guideDSADeepSeek Sparse Attention: a sparse-attention lane where routing and masking logic must stay faithful to the score path without breaking runtime constraints such as CUDA graph capture.GroundingAbout: DSA and CUDA graph safety History: DSA index cache patch Example: DSA CUDA graph safety sample CUDA-graphQuick term guideCUDA GraphsCUDA's capture-and-replay execution model, where hidden host sync points or Python-side branching break an otherwise valid GPU work graph.GroundingAbout: DSA and CUDA graph safety Example: DSA CUDA graph safety sample Example: CUDA graph block validation sample safety fix is carried behind feature detection so it disappears automatically once we move to an upstream version that includes the fix. The DSAQuick term guideDSADeepSeek Sparse Attention: a sparse-attention lane where routing and masking logic must stay faithful to the score path without breaking runtime constraints such as CUDA graph capture.GroundingAbout: DSA and CUDA graph safety History: DSA index cache patch Example: DSA CUDA graph safety sample memory rewrite stays local until the corresponding upstream implementation is available. The fused cross-entropy issue is covered by a thin integration wrapper that avoids the broken reduction shape. The Mamba-3 grouped-backward fix lives in a small local fork until upstream supports the middle grouping case directly. The dtype-repair logic is handled as a narrow integration shim rather than a broad global override.

That split matters operationally. Not every "upstream bug" from the same morning retires the same way. Some failures become source patches, some become compatibility gates, and some stay as version-floor or environment constraints until the owning project settles. Keeping those fix classes separate is why this incident log pairs with How we keep a patch lane and External library glitches we fixed instead of collapsing everything into one generic patch story.

The implementation details vary, but the rule does not: every local fix must say when it can be deleted.

Ablations and what we kept

The one ablation we care about on bugs like these is simple: does the workaround change the numerics?

The DSAQuick term guideDSADeepSeek Sparse Attention: a sparse-attention lane where routing and masking logic must stay faithful to the score path without breaking runtime constraints such as CUDA graph capture.GroundingAbout: DSA and CUDA graph safety History: DSA index cache patch Example: DSA CUDA graph safety sample graph-capture fix is behavior-preserving by construction. The DSAQuick term guideDSADeepSeek Sparse Attention: a sparse-attention lane where routing and masking logic must stay faithful to the score path without breaking runtime constraints such as CUDA graph capture.GroundingAbout: DSA and CUDA graph safety History: DSA index cache patch Example: DSA CUDA graph safety sample memory rewrite changes execution order but not the intended computation. The grouped Mamba backward path is checked against the two supported edge cases. The fused cross-entropy workaround is validated against a non-fused reference on the masked-loss case it replaces.

Operationally, what we kept was a disciplined patch lane: one entry per bug, one reproducer per entry, one readable explanation, and one retirement condition. What we dropped was the instinct to "just wait" for upstream on a bug that is blocking the current run. A patch lane is cheaper than a lost training day, and a clean upstream submission is cheaper than a patch that lives forever.

The broader lesson is that actively developed systems break in small, specific ways. None of these bugs was catastrophic in isolation. Each one costs somewhere between thirty minutes and a few hours if you debugQuick term guideDebuggingA grounded guide for debugging Modal failures in MegaCpp: cold starts, multi-GPU hangs, image drift, detached collector issues, and volume or…GroundingModal Debugging Guide for Training and Benchmark Failures OOM Debugging Playbook for H200 Training Runs it cold, and far less if you already have a patch lane ready to absorb the fix. The savings are not only in bug-finding. They are in removing hesitation about whether to patch locally, how to document it, and how to retire it later.

The patch lane is a filter, not a dump. A local patch exists to keep today's run green. A finished upstream contribution exists to stop the patch from living forever. Between them sits the real work: a reproducer we trust, an explanation someone outside the team can review, and a check that the fix is not already in flight upstream.