External library glitches we fixed

Running a hybrid Mamba-3 plus DSAQuick term guideDSADeepSeek Sparse Attention: a sparse-attention lane where routing and masking logic must stay faithful to the score path without breaking runtime constraints such as CUDA graph capture.GroundingAbout: DSA and CUDA graph safety History: DSA index cache patch Example: DSA CUDA graph safety sample recipe on a nightly PyTorch stack means every library in the fan-out is moving while we use it. This post is the public catalog: grouped by library, each entry names the symptom, our local workaround, and the upstream contribution we prepared or landed. The goal is not to expose every non-public breadcrumb. It is to show the engineering pattern clearly enough that another team could apply it.

Why MegaCpp cares about this

A well-kept patch lane is one of the cheapest forms of institutional memory we have. When a training run goes NaN at iteration 3, we do not want to rediscover a bug we already fixed two weeks ago under a slightly different shape. The rule we enforce is simple: every local patch must point either to an upstream issue or PR, or to a public-facing draft that is ready to become one. Every retirement of a local patch must be explicit.

That patch-lane stance is easier to see next to one morning of bugs and upstream PRs overview, which show the same work from the incident and filing sides.

What we built in practice

PyTorch and Dynamo

We currently carry no source patches against torch itself. What we do carry is a narrow set of Dynamo configuration choices wired in early, together with explicit torch._dynamo.disable boundaries around kernels and routing paths that are not yet good compile citizens.

The load-bearing workaround here is configuration, not source divergence. One real torch-side glitch we still account for is an older reduce_scatter_tensor regression that showed up in Megatron tests on earlier versions. The fix there is a version floor, not a patch.

torch_xla and libtpu

The TPU lane has its own class of regressions and does not overlap much with the GPU incidents in this catalog. The shared discipline is provenance: every TPU run records the exact torch, torch_xla, and libtpuQuick term guidelibtpuThe TPU backend library that pairs with PJRT/XLA and owns device-side execution underneath the frontend.GroundingAbout: libtpu / PJRT ownership boundaries Example: TPU backend ownership note Example: XLA runtime probe sample combination so a later bisect can answer whether an XLA update changed behavior.

We did not need to carry fresh upstream code patches from that lane in this window. The practical answer was rolling back to a known-good set of pins and keeping the validation trail clean. That is the same distinction expanded in Torch/XLA PJRT reality: a version pin is part of the runtime receipt, not automatically part of the long-term source patch lane.

NCCL and collectives

Most NCCLQuick term guideNCCLNVIDIA's collective-communication library for all-reduce, all-gather, reduce-scatter, and point-to-point transport on CUDA multi-GPU lanes.GroundingAbout: NCCL and collective hangs Example: pipeline parallel sample Reference: training on 8x H200 failures we see are topology or environment problems rather than NCCLQuick term guideNCCLNVIDIA's collective-communication library for all-reduce, all-gather, reduce-scatter, and point-to-point transport on CUDA multi-GPU lanes.GroundingAbout: NCCL and collective hangs Example: pipeline parallel sample Reference: training on 8x H200 source bugs. We carry no NCCLQuick term guideNCCLNVIDIA's collective-communication library for all-reduce, all-gather, reduce-scatter, and point-to-point transport on CUDA multi-GPU lanes.GroundingAbout: NCCL and collective hangs Example: pipeline parallel sample Reference: training on 8x H200 source patches.

One useful structural note belongs in this catalog anyway. A particular pipeline-plus-expert-parallel combination can deadlock because the topology assumptions of the pipeline schedule and expert synchronization do not line up. That looks like a low-level collectives failure until you understand the layout. The fix is not a source patch. It is an explicit compatibility gate and clear documentation. That is the failure family described more directly in NCCL and collective hangs: some "library bugs" are really invalid topology combinations that need a hard gate, not a local source fork.

Megatron-Core

Megatron-CoreQuick term guideMegatron CoreThe NVIDIA framework surface MegaCpp ports into through narrow adapters, layer specs, and runtime ownership bridges.GroundingAbout: Porting to Megatron friction About: Nemotron-style recipe as pure Megatron CLI Example: Mamba3 TP mixer sample is the largest section of the catalog because it sits on several critical boundaries at once: graph captureQuick term guideCUDA GraphsCUDA's capture-and-replay execution model, where hidden host sync points or Python-side branching break an otherwise valid GPU work graph.GroundingAbout: DSA and CUDA graph safety Example: DSA CUDA graph safety sample Example: CUDA graph block validation sample, fused loss paths, DSAQuick term guideDSADeepSeek Sparse Attention: a sparse-attention lane where routing and masking logic must stay faithful to the score path without breaking runtime constraints such as CUDA graph capture.GroundingAbout: DSA and CUDA graph safety History: DSA index cache patch Example: DSA CUDA graph safety sample implementation details, FP8Quick term guideFP8Eight-bit floating-point training and inference formats used to trade precision for throughput and memory on recent accelerator lanes.GroundingAbout: precision recipe: FP16, BF16, FP8, NVFP4 History: FP8 rollout notes Reference: Megatron FLCE on Hopper integration, and model wrappers.

The first class of issues involves CUDA-graphQuick term guideCUDA GraphsCUDA's capture-and-replay execution model, where hidden host sync points or Python-side branching break an otherwise valid GPU work graph.GroundingAbout: DSA and CUDA graph safety Example: DSA CUDA graph safety sample Example: CUDA graph block validation sample safety in DSAQuick term guideDSADeepSeek Sparse Attention: a sparse-attention lane where routing and masking logic must stay faithful to the score path without breaking runtime constraints such as CUDA graph capture.GroundingAbout: DSA and CUDA graph safety History: DSA index cache patch Example: DSA CUDA graph safety sample. We hit eager-style assertions and sentinel-handling branches that are harmless in normal execution and illegal during stream capture because they synchronize through .item(). The local fix was to gate or rewrite those checks so capture stayed legal. The upstream lesson is broader: graph-capture compatibility must be audited explicitly, even for code that looks like harmless validation. The public-safe checked-in chain is DSA CUDA graph safety sample, DSA CUDA graph safety nearcopy, CUDA graph block validation sample, and the filing discipline summarized in Upstream PRs overview.

The second class involves DSAQuick term guideDSADeepSeek Sparse Attention: a sparse-attention lane where routing and masking logic must stay faithful to the score path without breaking runtime constraints such as CUDA graph capture.GroundingAbout: DSA and CUDA graph safety History: DSA index cache patch Example: DSA CUDA graph safety sample memory behavior. One implementation materialized a large fp32 intermediate before reducing it, which was mathematically correct and operationally expensive. The local fix was to stream the reduction and accumulate directly into the smaller destination tensor. The checked-in shape receipt is the narrow one: an upstream-style [sq, b, h, sk] fp32 resident collapses to a streamed [b, sq, sk] accumulator in DSA indexer memory fix and DSA indexer memory checked-in example. The lesson is that many memory bugs are really "wrong intermediate" bugs.

The third class involves fused loss paths. We hit a fused linear cross-entropy backward path that behaved correctly for scalar-style reductions and incorrectly for non-uniform per-token weighting. That is the kind of bug that can hide in plain sight because the default tests do not exercise masked reductions. The local workaround was to keep the fused path where it was safe and route the unsupported reduction shape through a correct wrapper. The loss-boundary version of that bug family is documented in Megatron FLCE on Hopper.

We also hit architecture-gating issues around fused linear cross-entropy support, plus a regression where one model path no longer picked up the fused head that the adjacent GPT path already used. Those are easy bugs to miss because they often arrive as integration regressions rather than obvious math failures.

Triton

We did not carry first-party Triton source patches in this window, but we did hit Triton-adjacent behavior worth documenting. In one kernel, a math intrinsic lowered less efficiently than expected on current nightlies, so the local implementation kept a more explicit fast path.

That is not automatically an upstream bug. Sometimes the right move is to document the limitation, use the clearer local path, and wait until the compiler contract is stable enough to justify a contribution.

Transformer Engine

Transformer EngineQuick term guideTransformer EngineNVIDIA's Transformer Engine library path for accelerated Transformer modules and lower-precision training surfaces such as FP8, kept behind optional adapter seams in these posts.GroundingAbout: Transformer Engine on H200 and Blackwell-class GPUs: the bridge we use Reference: NVIDIA Transformer Engine documentation Reference: Transformer Engine FP8 and FP4 primer sits close enough to the critical path that it deserves its own entry. On one hardware lane we saw an FP8Quick term guideFP8Eight-bit floating-point training and inference formats used to trade precision for throughput and memory on recent accelerator lanes.GroundingAbout: precision recipe: FP16, BF16, FP8, NVFP4 History: FP8 rollout notes Reference: Megatron FLCE on Hopper backward path fail an backend alignment assertion even though the architectural dimensions looked valid. The same stack passed on a different accelerator lane.

The practical fix was to limit FP8Quick term guideFP8Eight-bit floating-point training and inference formats used to trade precision for throughput and memory on recent accelerator lanes.GroundingAbout: precision recipe: FP16, BF16, FP8, NVFP4 History: FP8 rollout notes Reference: Megatron FLCE on Hopper hybrid training to the lane that we had validated and keep the other lane on BF16 smoke coverage. Not every library issue should be patched immediately. Some should be isolated, documented, and routed around until the owning project can address them properly.

Fast Hadamard Transform

This was not a code bug, it was a packaging bug: an sdist was missing a core C++ source file. The source repository was fine, the published source package was not.

The workaround was operational rather than architectural: install from a known-good source instead of pretending the broken package belongs in the long-term patch lane. Packaging mistakes should usually live in bring-up scripts and environment notes, not in the same bucket as durable code fixes.

How it lands in MegaCpp

Each entry above has a mirror in production. Some live as small import-time overlays or subclass overrides in the Megatron integration layer. Some live in a narrow local mamba_ssm fork. Some remain operational notes because the right answer is a pin, a feature gate, or a hardware-specific restriction rather than a code patch.

The storage mechanism is less important than the discipline around it. Every local fix must be idempotent where possible, must be tied to a known upstream state, and must have a visible retirement condition.

The useful split is that not every dependency incident belongs in the same bucket. A source patch, a version floor, a topology gate, and a packaging-source pin can all resolve "the library broke," but they retire differently. We keep them separate so the patch lane only owns durable source divergence, while runtime pins, compatibility gates, and bad-package workarounds stay visible as the narrower operational constraints they actually are.

Ablations and what we kept

The ablation question for every entry is the same: does the workaround change training numerics in a way that matters?

The DSAQuick term guideDSADeepSeek Sparse Attention: a sparse-attention lane where routing and masking logic must stay faithful to the score path without breaking runtime constraints such as CUDA graph capture.GroundingAbout: DSA and CUDA graph safety History: DSA index cache patch Example: DSA CUDA graph safety sample patches are behavior-preserving by construction. The grouped Mamba fixes are checked against supported edge cases. Precision-related fixes are compared against higher-precision references. Fused-loss workarounds are validated against non-fused baselines on the masked-loss shapes that motivated them.

What survived contact with real hardware is in this catalog. What did not make it in were one-off hacks that only hid the symptom. Nothing should enter a public patch inventory unless it is readable, testable, and worth upstreaming.

Production checklist

• Every local patch names a corresponding upstream issue, PR, or public-ready draft.
• Every local patch has a retirement condition.
• Every patch ships with a reproducer that fails without the patch and passes with it.
• Import-time patches are idempotent and gated on the upstream state they compensate for.
• Regression guards stay alive after the upstream fix lands; deletion is explicit, not assumed.
• Packaging bugs live in environment bring-up notes, not in the long-term patch lane.
• Operational workarounds such as env flags or backend restrictions are tracked separately from real source patches.