How we keep a patch lane

Every serious training stack we run against nightly software needs a patch lane: a small set of local fixes, pinned environments, and a maintained inventory of what we carry and why. This post is the operational view: how we keep that lane honest while upstream code changes every week, and how we decide when a local diff can retire. The companion posts One morning of bugs and External library glitches we fixed cover the incidents themselves; this one covers the mechanism.

Why MegaCpp cares about this

MegaCpp depends on parts of the PyTorch ecosystem that move quickly: nightly PyTorch builds, current accelerator libraries, rapidly changing model code, and a TPU lane with its own version constraints. None of those surfaces stays stable for long. If we waited for every regression to be fixed upstream before training, we would stop shipping useful work. The patch lane is how we keep moving without letting temporary fixes harden into permanent drift.

The two rules that matter most are simple: every workaround must be reviewable, and every workaround must have a retirement condition. Drift accumulates quietly when either rule slips.

What we built in practice

Pinned environments are the foundation

The first layer of the patch lane is not code, it is provenance. We carry several environment bundles across different hardware lanes, and each one is pinned tightly enough that we can say exactly which upstream snapshot a fix was validated against.

Nothing here is "just install the latest wheel and hope." We install in a controlled order, avoid accidental dependency resolution, and log the active stack line for every meaningful run. That makes later bisects tractable. A patch note without a known environment is not much use.

Pins also need artifact identity, not just version strings. Fast-moving nightly stacks are too easy to poison with registry drift, accidental re-resolution, or a differently built wheel that happens to share the same nominal version. In practice that means mirrored artifacts, checksum-verified installs, and one clear precedence order for where packages are allowed to come from.

The same rule applies to public claims about sources, tokenizers, datasets, and benchmark inputs: the reader should be able to find the exact snapshot, not a floating label. The short version is in our reference corpus pinning notes, which keep revision, license, retrieval, and schema metadata together.

That is also why the lane separates lock creation from lock consumption. A lock update is an explicit maintenance event; ordinary reproducer and verification runs should stay on locked syncs so dependency drift fails fast instead of silently repairing itself during the run. If the lock and project metadata disagree, that is upgrade work and belongs in the same review wave as the matching patch-inventory update.

Local forks and overlays stay small on purpose

The second layer is a small, deliberate set of local forks or overlays. The rule is straightforward: only carry a fork when there is a real diff to justify it, and every carried change needs a retirement condition.

In practice that means a small mamba_ssm fork for a few targeted backward-path fixes, a TileLangQuick term guideTileLangA CUDA kernel DSL/compiler surface used here for explicit tile layouts, shared-memory legality fixes, and TMA-oriented kernel experiments.GroundingAbout: TileLang TMA and H200 reality History: upstream PR: TileLang and Megatron Example: TileLang TMA bulk-copy sample working tree for a handful of precision and dispatch issues, and a lighter Megatron overlay for surgical fixes that do not justify maintaining a large long-lived branch. We avoid carrying heavyweight forks of foundational libraries unless there is no cheaper path.

The important discipline is not the exact mechanism. It is that the local fix is narrow, understandable, and easy to remove once upstream catches up.

Overlays also need a strict boundary. If the change can stay Python-only behind an explicit import seam, an overlay can be cheaper than a fork. If it depends on global monkey-patching, or if it crosses into a compiled extension or C++ binding, the patch lane should stop being clever and carry a real diff instead. That keeps stack traces honest and makes retirement reviewable.

The patch inventory is the catalogue

The third layer is the checked-in catalogue. We keep one entry per issue, together with a reproducer, a short explanation, and the current upstream status.

patch inventory:
  issue notes
  status tracker
  focused reproducers
  validation metadata

Each entry is written in the shape of an upstream contribution: target project, problem, solution, changed surfaces, and testing evidence. Reproducers are kept small enough that an outside reviewer can run them against the pinned environment. Validation metadata records which entries were exercised end to end and which are still in preparation.

We do not use this catalogue as a diary. An entry is there because it can plausibly become an upstream contribution. If a workaround is too ad hoc to explain cleanly, that is usually a sign that the workaround itself is weak.

The inventory also needs to answer a second question at upgrade time: not just "does this patch still apply?" but "is it still ours?" A structured registry with upstream status and an explicit retire-or-keep decision makes dependency bumps tractable instead of turning every version change into archaeology.

The registry gets even cheaper to operate when each entry preserves three more fields explicitly: the exact files touched, the public upstream thread it is waiting on, and the current lifecycle state. That is what lets an upgrade pass sort the inventory into keep, conflict, or retire before humans start re-reading diffs by hand.

Regular upstream diffing keeps the lane honest

The fourth layer is a boring but necessary check: a regular diff against current upstream. It asks, for every local fork or patch, "what are we still carrying that upstream does not?" Humans review the result. That is how a patch lane stops turning into folklore.

That check has exactly three acceptable states:

1. Zero diff: upstream absorbed the change, so we can retire it locally.
2. Non-zero diff with a matching entry: expected and tracked.
3. Non-zero diff with no matching entry: process bug, stop and explain.

That third state catches the most expensive kind of drift: a hot fix that kept a run alive but never made it into the inventory.

The merge-back pass is stronger when it classifies every carried fix into keep, conflict, or retire and reruns the narrow reproducer that justified the patch in the first place. A diff that still applies is not enough; the real question is whether the patch still fixes a live problem on the current pin or whether upstream quietly absorbed it.

The other quiet win is a lightweight upstream-status sweep between version bumps. If a linked upstream fix lands mid-cycle, the inventory should surface that early so retirement happens as a small cleanup instead of as surprise archaeology during the next dependency jump.

Submission waves beat drive-by PRs

The fifth layer is social, not technical: we batch submissions. The early mistake was filing upstream PRs opportunistically whenever a local fix landed. Quality varied, context switching was expensive, and maintainers received a stream of half-polished patches.

Our current cadence is simpler and more respectful. On a regular schedule, someone reviews the inventory, checks the state of related upstream issues and PRs, reruns the upstream diff against the current pin, updates statuses, and only then decides which entries are ready to submit.

This sounds slow, but it is faster in aggregate. One clean, well-scoped submission does more good than several rushed ones that stall in review.

The cadence has a second benefit: it is when retirements actually happen. Without a recurring pass, "retire this once upstream merges" becomes a promise nobody keeps.

That batching only works because every candidate still carries a narrow reproducer when the wave starts. We do not submit because a diff still applies. We submit because the public-safe repro still fails on the current pin, still passes with the local patch, and still has an obvious retirement condition once upstream absorbs it. That is the same receipt discipline used in Verifier-first C++ evals and Compile-time vs runtime tradeoffs: keep the smallest artifact that proves the fix is still real.

How it lands in MegaCpp

Production MegaCpp inherits the patch lane, but the shape changes.

First, the import-time patch surface shrinks. A fast-moving research-stack can justify small feature-detected overlays; long-lived product code should move those decisions into clearer seams such as subclassing, configuration, or explicit integration points.

Second, the environment matrix narrows. Production ships fewer hardware lanes than research, and the difference between research and production stacks is tracked explicitly.

Third, release notes absorb the retirement trail. A production release should make it obvious which local patches disappeared and which upstream versions replaced them.

One practical rule from the research packs is worth stating plainly: when a fix crosses a low-level runtime seam and a higher-level Python binding, those pieces need to land together or not at all. Partial merges are how you create regressions that only show up after the version bump looks "done."

Ablations and what we kept

We tried three things that did not survive contact with real training.

We tried a single lockfile across CUDAQuick term guideCUDANVIDIA's GPU programming stack: compiler, runtime, driver, libraries, and kernel toolchain used by CUDA training and inference lanes.GroundingAbout: XLA vs CUDA stack decisions History: GB10 tensor-path proof summary Reference: training on 8x H200, TPU, and CPU. It was painful to produce, painful to update, and inaccurate often enough that it became a source of confusion rather than clarity. We dropped it in favor of explicit pinned environments.

We tried to carry every Megatron fix as a full fork. That worked, but rebasing against a moving development branch became too expensive for changes that were often very small. We kept fuller forks where the diff justified them and used lighter overlays elsewhere.

We also tried a strict upstream-first rule: never land a local patch until a polished upstream PR exists. That sounds principled, but on a fast-moving training stack it can waste an entire training window. We replaced it with a better rule: fix locally when needed, draft the upstream-quality explanation immediately, and submit when the patch is ready.

What we kept is the part that compounds: pinned environments, a small set of local forks or overlays, a public-facing patch inventory, a regular upstream diff, and a weekly merge-back cadence. We also kept three hard rules: every local patch must be tracked, every tracked patch needs a retirement condition, and nothing retires silently.

Production checklist

• Pinned environments are the source of truth. No lockfile heroics, and installs should not drift underneath you.
• Carry full forks only when the diff is substantial; use lighter overlays for genuinely small surgical fixes.
• Every local patch needs a matching public-facing entry with a reproducer and a readable explanation.
• Every entry needs a retirement condition and a clear upstream status.
• Regular upstream-diff checks should make undocumented drift impossible to ignore.
• Weekly merge-back reviews should update status and retire work that upstream has absorbed.
• Import-time patches should be idempotent and feature-detect the upstream shape they compensate for.
• Retirement should be explicit in version control, not tribal knowledge.
• Submission volume should respect reviewer bandwidth.