A curated archive for the C++ data path: corpus selection, semantic enrichment, packaging into training artifacts, and the file-level durability choices that keep the pipeline sane.
This cluster works best in sequence. Start with the broad pipeline and corpus articles, then move into the semantic graph and packaging layers that make the training rows useful.
Best if you want to understand where the C++ training rows come from and why the pipeline is intentionally shard-heavy.
These define what goes into the corpus and how the base rows are formed.
An honest walkthrough of how the MegaCpp training data pipeline was built — source selection, filtering, dedup, tokenization, document masking, and the quality gates that catch our own mistakes.
The broad retrospective: what worked, what broke, and how the pipeline settled.
A detailed walkthrough of how MegaCpp builds a C/C++ corpus: source selection, pins, deduplication, compile-command metadata, chunking, structure-aware exports, and refusal rules.
The keep-versus-drop policy for raw C/C++ source and why the filtering matters downstream.
Every stage of the MegaCpp data preparation pipeline: ingest, dedup, license filtering, document masking, tokenization, packed rows, and the checks that keep dataset snapshots trustworthy.
The end-to-end article for the shaping and movement of training rows before packaging.
The structural metadata layered on top of raw C++ source: structure IDs, chunk boundaries, call edges, type edges, tree-sitter AST features, and the optional libclang semantic graph. What each one is for, what the ablations justified, and what we pay in storage and runtime.
The shortest explanation of which enrichment layers were kept in the pipeline and why they survived the cost review.
How MegaCpp deduplicates C++ at scale: shingling choices, MinHash/LSH parameters, exact-dup SHA-256, and the tradeoffs behind near-duplicate removal.
The deduplication-side companion once the raw corpus is big enough that redundancy becomes a quality and cost problem.
These explain why the pipeline costs more than plain text ingestion and why that cost is accepted.
How compilation-database-driven semantic extraction improves C++ corpus quality, where clang indexers fail, and why build-aware graphs matter more than raw text proximity.
Why real build context is required for C++ training quality rather than optional metadata decoration.
How the libclang-based semantic indexer feeds v6_enriched parquet: compilation-database handling, the per-file translation-unit graph, call and type edges, the failure modes we hit, and the wall-clock cost of ground-truth semantics.
The most direct explanation of the semantic indexer, its perf wall, and what it buys the stack.
Why MegaCpp keeps a narrow data bridge from tokenized parquet shards to Megatron indexed datasets instead of tying data preparation to one runtime import surface.
The bridge from prepared training shards into the indexed dataset artifacts the training lane actually consumes.
Why a parquet-to-binidx bridge matters, what contract it has to preserve, and why a thin formatting wrapper is worth keeping separate from the low-level converter.
How prepared shards are turned into the Megatron-friendly artifact format used by the training lane.
Why packed rows are the real boundary between the data pipeline and the model, and why MegaCpp treats row packing as a schema contract rather than a storage detail.
The most direct description of why packed rows became the real training contract instead of a minor loader detail.
These complete the archive with the file-level contracts that keep the data lane reproducible.
Why large-message serialization becomes fragile near protobuf's practical limits, and how MegaCpp's checkpoint and data paths avoid single huge payloads by using sharded files, streaming conversion, and explicit completion markers.
A good durability companion piece: why the pipeline keeps choosing shards and markers over giant blobs.
Why schema discipline, canonical fallback values, and explicit versioning matter more than format churn when a C/C++ training corpus gains structure-aware metadata.
The schema- and versioning-side document once the enriched rows have to stay compatible across pipeline passes.
A detailed walk through every schema generation of the C++ training corpus - what each version added, the schema diff, the storage cost, the val_bpb delta we attribute to each step, what we deprecated and why.
A useful explanation of how dataset version drift shows up in practice instead of only in high-level release notes.
How MegaCpp describes source provenance, revision pinning, SPDX metadata, and refusal-list rules for a public C/C++ corpus narrative without overstating legal certainty.
The provenance companion piece for why the data lane keeps explicit records around source, license, and recoverability.
Keep exploring
These hubs cover nearby parts of the blog without turning the archive into a giant taxonomy.
A curated GB10 and Blackwell reading path: consumer-versus-datacenter tensor paths, driver-visible false positives, arch-patch repros, and the serving or precision choices that survived contact with the hardware.
A curated Modal reading path: when the rented-GPU surface was useful, what broke on multi-GPU launches, how receipts were recorded, and how we kept the lane debuggable.
A curated Mamba3 reading path: why MegaCpp kept a hybrid stack, how the kernels evolved across CUDA, TileLang, and TPU, and where the runtime wins actually held.
A curated MLA reading path: the weight-absorption contract, Megatron-safe integration boundaries, dispatch and FP8 edges, and the adapter surfaces that keep MLA connected to the rest of the stack.
A curated evaluation reading path: verifier-first harnesses, ablation structure, benchmark receipts, and the evidence rules that keep comparisons from collapsing into anecdotes.
A curated Megatron reading path: the parallelism map, what actually splits, how NVIDIA and TPU wrappers differ, and the migration surfaces around NAM56R-style layouts.
A curated TPU sparse-attention reading path: block-sparse contracts, Pallas kernel choices, SPMD sharding, and the runtime surfaces that keep long-context TPU work stable.
A curated path through the H200 lane: operator bring-up, step-time anatomy, memory pressure, and the NVIDIA kernel surfaces that actually moved the stack.
A curated reading order for TPU work: bring-up, PJRT and Torch/XLA boundaries, SPMD sharding, and the kernel/runtime traps that made TPU performance non-obvious.
A curated path through the expert stack: what the specialist path changed, how routing works, and how the parallelism map constrains the model layout.