Disclosure: I used AI assistance to edit and refine this post, but the ideas, interpretations, and conclusions are mine.

Part 1 ended with a table of eight things I expected to see. I had not rented a GPU at that point. The whole list came out of reading NanoChat and working through the arithmetic on paper.

This post is what happened when I ran it.

The short version: the model trained for 196.49 minutes on four H100s at 59.79% MFU, reached a validation bpb of 0.7194, and the instance bill came to about $41. Seven of the eight expectations held. The eighth held too, but only once I was honest about which step I was measuring.

What I did not expect was how quickly it would be settled. I had imagined careful post-hoc analysis. In practice, four of the eight predictions were resolved by line 373 of the log, before a single training step had run.

—

Making the expensive part boring

The one thing I was sure of before starting: I did not want to be learning cloud plumbing while a GPU meter was running.

Renting four H100s means the clock starts the moment the instance boots and does not stop for mistakes. So everything that could be rehearsed on cheap hardware got rehearsed there, and everything that could be automated got automated. The design goal was that the expensive run should be the least interesting part of the day.

Three decisions carried most of that weight.

The container holds code, not data. The Vast image was built with the runtime and the repository, nothing else. Datasets and checkpoints are large, they change independently of the code, and baking them in would have meant rebuilding the image for every data change. Instead the container starts, brings up SSH, and idles until told what to do.

Generated artifacts sync to Google Drive; downloaded ones do not. Anything the run produced — tokenizer, checkpoints, logs, report — was pushed to durable storage in the background. Anything that could be re-downloaded from the source was not. This distinction matters more than it sounds: a fresh host could restore the expensive, irreplaceable things in under two minutes and then re-fetch the cheap, reproducible ones in parallel.

Orchestration ran locally. A local script drove the remote host, which meant the workflow was scriptable, resumable, and legible to an agent. Each stage wrote a status marker — STARTED, DONE — so progress was a file to read rather than a terminal to watch.

I will not pretend this was elegant on the first attempt. It took several iterations, and the iterations are why the final run was uneventful.

Three tiers, one of which vanishes. Restore runs once and moves very little; sync runs throughout and moves everything the run produces. Losing the host costs the hours since the last sync, not the run.

Everything before the run

Before any H100 was involved, the pipeline ran end to end on two RTX 3060s at depth 12. That configuration produces a bad model. That was never the point. The point was that every stage — tokenizer, base training, evaluation, SFT, chat evaluation, report generation — executed in order, wrote what it was supposed to write, and synced what it was supposed to sync.

Cheap hardware catches the whole class of failures that have nothing to do with scale: wrong paths, missing environment variables, a sync loop that silently uploads nothing, an evaluation stage that cannot find the checkpoint the previous stage wrote.

Then production blew up anyway.

The failures on H100 hardware were not code failures. They were operational: an SSH proxy that would not hold a connection, instances that idled while I worked out why. The billing screenshot from that day shows the debris — several instances at $5.32, $0.82, $0.17, $0.10, and two under a cent. None of them trained anything.

This is the part of the story that does not flatter anyone, so it is worth stating plainly: the code was never the problem. The problem was me learning someone else’s infrastructure at $2 an hour.

The run

The final run was tagged d24-4xh100-full: depth 24, four H100 80GB cards, FP8 enabled, started 2026-07-11 at 12:05:16.

Setup took about five minutes. Google Drive restore finished in 1.68 minutes, recovering the shared tokenizer and confirming there were no run-specific checkpoints to resume from. The dataset download then pulled 171 shards while the tokenizer was reused rather than retrained.

At 12:10:07 the training script initialised, and the log started answering questions.

Four predictions in forty lines

I had expected to do this analysis afterwards, with the log open in one window and Part 1 in another. Instead the initialisation block resolved four of the eight expectations before the first step.

Flash Attention 3. Line 342:

✓ Using Flash Attention 3 (Hopper GPU detected), efficient, new and awesome.

Expectation 5, settled. The Hopper gate worked exactly as the import chain said it would.

The depth cascade. Lines 343 to 368 printed the model configuration and the derived hyperparameters:

"n_layer": 24, "n_head": 12, "n_embd": 1536, "window_pattern": "SSSL"
Auto-computed optimal batch size: 1,048,576 tokens
Scaling LRs by 1.4142 for batch size 1,048,576 (reference: 524,288)
Scaling the LR for the AdamW parameters ∝1/√(1536/768) = 0.707107
Calculated number of iterations from target data:param ratio: 5,568
Total number of training tokens: 5,838,471,168

Expectation 1 was that --depth=24 would produce exactly these numbers. Every one matches what I derived from the formulas in Part 1. model_dim 1536, 12 heads, batch 1,048,576, LR scale 1.4142, AdamW scale 0.707107, 5,568 iterations.

The parameter table three lines above makes the derivation auditable in a way I had not anticipated. It reports transformer_matrices at 679,478,976 and lm_head at 50,331,648. Those sum to 729,810,624 — the scaling-parameter count the iteration formula depends on, printed as its own components rather than asserted as a total.

The batch math. Lines 371 to 373:

Tokens / micro-batch / rank: 16 x 2048 = 32,768
Tokens / micro-batch: 131,072
Total batch size 1,048,576 => gradient accumulation steps: 8

Expectation 2 was that running on four GPUs instead of the reference eight would change the accumulation count and nothing else. There it is: 1,048,576 ÷ 131,072 = 8. The global batch is a property of the experiment; the world size is a property of the hardware I happened to rent. The code keeps them separate, and says so out loud.

FP8 eligibility. Line 354:

✓ FP8 training enabled (tensorwise scaling) - converted 145/158 linear layers, skipped 13 (too small)

This is the one I had been most pleased with in Part 1, and the one I was most nervous about. I had counted the linear layers by hand from the module definitions, applied the eligibility filter from fp8.py, and predicted 145 conversions and 13 skips — twelve ve_gate projections at Linear(12, 12), where 12 is not divisible by 16, plus one smear_gate at Linear(24, 1).

The log prints 145/158 and 13. Expectation 3, exactly.

Lines 342 to 373 of the run log, abridged. Each printed value is tied to the Part 1 expectation it settles. Four of them resolve before step one, because the script says what it decided on its way up.

Watching it run

With four expectations already resolved, the remaining ones needed the run to finish. So I watched the step lines scroll.

They are almost boring, which turns out to be the finding:

step 00001/05568 | loss: 10.371374 | dt: 2108.35ms | tok/sec: 497,344 | bf16_mfu: 60.03
step 00002/05568 | loss: 10.325244 | dt: 2111.80ms | tok/sec: 496,531 | bf16_mfu: 59.94
step 00003/05568 | loss: 10.253621 | dt: 2117.46ms | tok/sec: 495,205 | bf16_mfu: 59.78

Step 1 is already at 60.03% MFU. Not step 500, not after some warm-up ramp. The second step of training.

The receipts that needed the whole run

Utilisation, honestly

Expectation 7 was the one I cared most about, and the one most easily fudged. The claim was not that MFU would be high. It was that it would be stable — that the headline number would not be a decent average hiding a bad distribution.

A single reported figure cannot answer that, so I pulled all 5,568 step lines out of the log and computed the distribution.

Here is where I have to be careful, because the honest version of this claim is narrower than the flattering one.

Step 0 runs at 0.92% MFU. It is the compile step: torch.compile traces and builds kernels, which takes 137 seconds against roughly 2.1 seconds for every subsequent step. Including it in a stability measure inflates the standard deviation nearly threefold, from 0.32 to 0.85, on the strength of one sample that is not measuring steady-state throughput at all.

So the claim I will defend is the narrow one: across 5,567 steady-state steps, mean MFU was 59.66% with a standard deviation of 0.32, and not one of them fell below 50%. 98.6% of steps land within a single percentage point of the mean. Step 0 is excluded deliberately, and I am telling you that rather than quietly dropping it.

There is a detail I did not predict and would not have thought to look for. Splitting the run into quarters:

Utilisation does not sag as the run progresses. It drifts upward, monotonically, and the final quarter’s mean is 59.79% — which is the headline MFU the report quotes. The number in the report is not a flattering peak. It is where the run settled.

Every steady-state step, in half-point bins. 98.6% of them land within one point of the 59.66% mean. Step 0 sits at 0.92% and is drawn in its own lane so it cannot flatter the scale. The quarterly means rise and finish on the reported 59.79%.

What the hardware reported about itself. Everything above is the training script’s own arithmetic: bf16_mfu is a number NanoChat computes and prints. WandB also samples the GPUs directly through NVML, on its own schedule, independent of the training loop. It is the only instrument in this run that is not NanoChat marking its own homework.

Six NVML panels for all four GPUs. Top row: enforced power limit flat at 700W, memory allocated flat at 60 GB and 70.2%. Bottom row: memory-access time, temperature, and utilisation, each showing the same regular sawtooth.

Utilisation sits at 99–100% for essentially the entire run. That rules out something I had not thought to check: host-side starvation. If the data loader had ever fallen behind, or if Python had opened gaps between steps, the GPUs would have gone idle and it would show here as a dip. Across 3.27 hours, they did not.

The regular dips are my favourite detail in the whole run. Every nine to ten minutes, utilisation falls, memory traffic drops away, and all four GPUs cool by several degrees before climbing straight back on the next sample. That is the validation pass: the log runs one every 250 steps, which is 8.85 minutes of training time, and the pass itself accounts for the rest of the gap. There are three deeper drops as well, and the run contains exactly three full CORE evaluations — though the log excludes evaluation time from its own clock, so I could not line the timings up closely enough to call that proven. The regular sawtooth I am sure of. You can read the evaluation schedule off the temperature of the silicon.

The flat panels are quietly reassuring in a different way. The enforced power limit holds at 700W throughout, so these were full-power SXM cards that were never quietly capped. Temperatures sit between the mid-fifties and low seventies, nowhere near throttling. Memory allocation climbs to 60 GB about four minutes in and never moves again — 70.2% of each card, which implies 79.6 GiB of total capacity and confirms these were the 80 GB H100s I thought I had rented. Flat for 5,568 steps, 24 validation passes and 3 full evaluations also means nothing leaked.

One thing this does not settle, and I want to be explicit because the chart invites the wrong conclusion. Utilisation counts time with a kernel resident on the device, not time doing useful work, and NCCL collectives are kernels too. A run whose communication blocked would look much the same here. This corroborates the flat MFU; it does not prove the overlap.

The arithmetic that proves there is no padding

Expectation 4 was about the data loader: that BOS-aligned best-fit packing fills every row exactly, with no padding anywhere.

This one has a proof rather than a measurement, and it is my favourite thing in the log.

The run reports 5,568 iterations and a global batch of 1,048,576 tokens. It separately reports the total tokens trained as 5,838,471,168.

5,568 × 1,048,576 = 5,838,471,168

Exactly. No remainder, no rounding, no discrepancy of a few thousand tokens that would indicate a padded final row or a dropped partial batch. Every token counted in the total is a real token that contributed to a gradient.

That is what “no padding, ever” looks like when you can check it with multiplication.

Half the GPUs, twice the time

Expectation 6 was that base training would be largely GPU-bound, so running on four cards instead of the reference eight should take roughly twice as long.

NanoChat’s README reports 1.65 hours for base training on an 8×H100 node. My run reports:

Total training time: 196.49m

That is 3.27 hours. Against 1.65, the ratio is 1.98×.

Two percent off perfect linear scaling, on half the hardware. I want to be careful about what this does and does not show: it is one data point at one configuration compared against a published number from a different machine, not a scaling study. But it is consistent with the design reading from Part 1 — that this path is single-node and compute-bound, and that the communication is arranged so it does not become the limit.

What it cost

The successful instance shows $41.16 on the Vast.ai billing page for 11 July. Including the smoke tests and the failed H100 attempts from the same day, the practical total was about $47.58.

The Vast.ai charges. The successful run sits alongside the debris from the attempts that did not get there, which is the honest version of the number.

The $100 line from the original GPT-2 speedrun framing is comfortably clear either way, and that was expectation 8.

The honest caveat: I am inferring which billing row corresponds to the final run from the timing and the amount. The instance identifiers are not stamped in the training log. The $41.16 row is by far the most likely candidate — it is the only charge consistent with a four-hour four-GPU session — but I have not proved the mapping, and I would rather say so than present it as certain.

The part of the bill that is not GPU time

One operational detail worth surfacing, because it surprised me.

A checkpoint is not one file. Rank 0 writes the model and metadata; every rank writes its own optimizer shard. After the run, the Google Drive metadata showed:

• Base checkpoint set: 9.283 GiB across 6 objects
• SFT checkpoint set: 9.283 GiB across 6 objects
• Model file model_005568.pt: 4,227,935,530 bytes
• Each of four optimizer shards: 1,434,912,917 bytes

My naive estimate had been 2 bytes per parameter for bf16 weights — about 2.77 GB for 1.38 billion parameters. The actual model file is over 4 GB, because a checkpoint carries more than a flat dump of parameter bytes.

This is why artifact sync ran continuously in the background rather than as a final step. Roughly 18.5 GiB of checkpoints moving to durable storage at the end of a run is not a rounding error on a rented machine. Getting that wrong means paying for GPU time while a network transfer finishes.

Talking to it

None of the above is why anyone trains a model.

After SFT and chat evaluation, there is a model you can hold a conversation with. It is not good. It is a 1.4-billion-parameter model trained on 5.8 billion tokens for three hours, and it will confidently tell you things that are not true.

Fluent, well formed, on topic, and wrong about things it has no way to know. Both halves of that sentence matter.

But it answers in fluent, well-formed English, follows the shape of a conversation, and stays on topic. Its ChatCORE score is 0.3712 and its CORE is 0.2561, against 0.2626 for the reference 8×H100 run — close enough that the difference is not the story.

The story is that the thing talks back, and it cost $41.

The tokenizer joke

One failure is worth keeping, because it is funny and because the lesson is real.

At one point I loaded a checkpoint with the wrong tokenizer. The model was fine. The weights were fine. Every token index was being decoded against the wrong vocabulary, so the output was confident, fluent, perfectly-structured nonsense.

The same checkpoint with a mismatched tokenizer. It has the cadence and punctuation of language right up until you try to read it.

It is a good reminder that a checkpoint is not self-describing. Weights plus the wrong vocabulary is not a degraded model; it is a different one. Ship the tokenizer with the checkpoint, or you have shipped an artifact that cannot be used.

What held up, and what I never tested

Eight expectations. Here is the accounting.

Now the part that matters more than the table.

I did not validate the optimizer design. Part 1 argued that moving communication into an explicit async optimizer phase is what keeps utilisation flat. Flat MFU and pinned GPU occupancy are both consistent with that, and neither can distinguish it from communication that blocks, because the collectives occupy the device either way. I have no profiler trace, no comparison against a DDP-wrapped baseline, and no measurement isolating the communication phase. The claim remains a reading of the code that the results do not contradict.

I did not measure the attention windows. SSSL is confirmed in the config, and the code says 18 of 24 layers see only 512 tokens. I never instrumented attention to watch it happen.

I did not reproduce the dataset comparison. NanoChat’s claim that ClimbMix cut the reference run by 27% over FineWeb-EDU is the repository’s measurement, not mine. I ran one dataset once.

I did not test the scaling laws. Same boundary as Part 1. I confirmed the code implements what it claims; whether D^0.383 is the right exponent needs sweeps I did not run.

One run does not establish reproducibility. It establishes that on 11 July 2026, on this hardware, the code did what a careful reading said it would.

What I actually learned

I went in expecting to check a stack. What I ended up checking was whether a piece of engineering means what it says.

The satisfying part was not that the predictions held. It was how they held. The log does not report a batch size; it reports the micro-batch, the per-rank tokens, and the resulting accumulation count, so you can verify the arithmetic yourself. It does not report a parameter count; it reports the components, so the scaling-parameter figure the iteration formula depends on can be reconstructed. It does not claim no padding; it prints two numbers whose product proves it.

That is a deliberate choice, repeated often enough that it cannot be accidental. The code is written to be checked. Someone decided that a reader who wanted to verify a claim should be able to, and then did the unglamorous work of printing the intermediate values that make verification possible.

Reading the code was rewarding. Watching it print exactly what the reading predicted — four times over, in forty lines, before training started — was better.

The model it produced is small and often wrong. The engineering that produced it is neither.

Notes and sources

• Run: d24-4xh100-full, 4× NVIDIA H100 80GB HBM3, 11 July 2026. The report and the full speedrun.log are in my fork, under blog/evidence/part-1.
• MFU statistics computed from all 5,568 base-training step lines in speedrun.log. Step 0 excluded from steady-state figures, as stated above.
• GPU telemetry is sampled by Weights & Biases through NVML, outside the training loop. It is not something NanoChat logs.
• Reference comparison figures (1.65 h, CORE 0.2626) are from NanoChat’s own README leaderboard row, not a run of mine.
• Andrej Karpathy, NanoChat.

Disclosure: I used AI assistance to edit and refine this post, but the ideas, interpretations, and conclusions are mine.

Read the repo alongside this. Everything below is a close reading of a specific codebase, and it will land far better if you have NanoChat open in another window. It is small enough to skim in an evening. Every claim here links to the exact file and line it came from — follow a few of them, and disagree with me where you find something I misread.

I have read a lot about transformers. Books, papers, tutorial series, the usual conference talks. Before this I would have told you I understood how they were trained.

Then in early 2026 I sat down with Andrej Karpathy’s NanoChat and read it properly — not skimmed, read — and learned more in a fortnight than I had from any of it. Not because the code is clever in a way that shows off. Because it is simple in a way that clearly took enormous effort to arrive at, and because almost every line turns out to be load-bearing once you ask why it is there.

NanoChat describes itself as a simple experimental harness for training LLMs on a single GPU node: tokenization, pretraining, fine-tuning, evaluation, inference, and a chat UI. It is small enough that a motivated reader can hold the whole thing in their head. That is the trick, and it is not an accident.

This post is the first half of a pair, and deliberately the half with no results in it. Here I read the code and write down what I expect to happen when it runs. Part 2 rents four H100s and checks.

That order matters. It is easy to explain why something was fast after watching it be fast. It is harder, and more honest, to commit to the prediction first.

One note on lineage: NanoChat’s README credits modded-nanogpt for the speedrun framing and borrows some implementation from it, but they are not the same benchmark — modded-nanogpt races to a target validation loss, while NanoChat’s speedrun spans tokenizer through chat evaluation. Ancestry, not comparison.

The shape of the repo

The executable spine is runs/speedrun.sh, and it is refreshingly literal. In order: set up the Python environment with uv, download data, train and evaluate the tokenizer, run base pretraining under torchrun, run base evaluation, download identity conversations, run SFT, run chat evaluation, generate the report.

You can read the project’s entire intent off that one file. There is no orchestration framework, no config hierarchy, no plugin system. The pipeline is a shell script because a shell script is sufficient.

That has a consequence worth stating early: the wall-clock time and the final bill do not measure raw pretraining throughput. They measure the cost of producing a usable artifact and a report about it. Those are different numbers, and the second one is the honest one.

One scope note. The reference speedrun path runs SFT and chat evaluation but not RL. NanoChat does ship scripts/chat_rl.py, but it sits outside the path I followed.

The whole project on one page. Bar length is the wall-clock time each stage actually took in my run, and the lines show which library modules each stage touches. Everything after this section is a zoom into base_train.

One integer, seven decisions

Here is the design choice that made me pay attention.

NanoChat does not ask you to tune anything to get started. The README describes a single complexity dial — the depth of the transformer — and scripts/base_train.py means it literally. From --depth, the script derives model width, attention head count, parameter count, training horizon, global batch size, learning-rate scaling, and weight decay.

The mechanism is a reference model. The d12 configuration is treated as an empirically measured anchor: its compute-optimal token horizon and batch size come from NanoChat’s own sweeps, recorded in dev/LOG.md. Every other depth is extrapolated from that anchor using scaling-law relationships plus a few hardware constraints.

The chain runs like this:

depth
  → base_dim     = depth × aspect_ratio
  → model_dim    = base_dim rounded up to a multiple of head_dim
  → num_heads    = model_dim / head_dim
  → scaling_params
  → target_tokens  = ratio × scaling_params
  → total_batch_size = B_REF × (target_tokens / D_REF)^0.383
  → learning-rate and weight-decay scaling

No single step here is exotic. What is unusual is the discipline of routing all of them through one user-facing number — and then printing the derivation so you can check it.

The rounding step shows the layering. base_dim is depth × 64, which for depth 24 gives 1536, then rounded up to a multiple of head_dim — which for 1536 does nothing. The constraint is there anyway, because attention heads must divide evenly and FA3 wants head dimensions it can tile. A depth producing an awkward width gets quietly nudged to a workable one rather than failing later inside a kernel.

The full derivation, with the paper behind each rule, is in Appendix B. The short version: batch size follows Power Lines, scaling roughly as D^0.383; learning rate follows standard square-root batch scaling; weight decay follows the T_epoch framework, keeping B / (η × λ × D) approximately constant.

One integer fixing everything below it. d12 is the measured anchor, which is why its scaling factors are exactly 1.0; the d24 column is the configuration I planned to run, derived by hand from the formulas before I ran anything.

I reproduced this derivation by hand for depth 24 before running anything, which gives Part 2 its first thing to check.

What I expect to see — Running --depth=24 should print model_dim 1536, num_heads 12, a global batch of 1,048,576 tokens, an LR scale of 1.4142, an AdamW LR scale of 0.707107, and 5,568 iterations. If any of those differ, I have misread the code.

The model it builds

The model in nanochat/gpt.py is a compact GPT-style transformer, and I am going to assume you know what that means. Rotary embeddings, RMSNorm, QK normalization, ReLU-squared MLPs, no biases, untied input and output embeddings. Individually, each of these is a choice you can find justified in a paper somewhere.

More interesting is the overall shape of the forward pass:

token embedding
  → previous-token smear
  → transformer blocks
  → optional mid-layer backout
  → final norm
  → lm_head

The two unfamiliar names are both small and cheap. The smear mixes a little of the previous token’s embedding into the current one through a learned gate — bigram-like information for almost no compute. The backout subtracts a cached mid-layer residual before the final norm, removing low-level features the output layer does not need. Neither is a headline idea. Both are the kind of thing you add after watching a lot of training curves.

--depth sets the number of repeated Block modules, and the rest of the model shape follows from the cascade above. The parameter-by-parameter breakdown is in Appendix A.

The window pattern, and the one thing the comments get wrong

The detail I did not expect is in sliding-window attention.

NanoChat tiles a pattern string across layers. The default is SSSL, where L is full context and S is shorter. Reading _compute_window_sizes:

short_window = -(-long_window // 4 // 128) * 128  # ceil to FA3 tile size

At sequence_len = 2048 that is 512 — a quarter of the context, rounded to a tile boundary. So with SSSL tiled across 24 layers, layers 3, 7, 11, 15, 19 and 23 get full context, and the other eighteen see 512 tokens.

Eighteen of twenty-four layers never look further back than a quarter of the sequence. Six layers carry every long-range interaction in the model. That is a far more aggressive bet than the phrase “sliding-window attention” suggests, and a large part of why the model is cheap to train.

A final safety line, window_sizes[-1] = (long_window, 0), forces the last layer to full context whatever the pattern says. For SSSL at depth 24 it changes nothing — layer 23 is already L. It only bites on a pattern ending in S.

This is also the one place I found where the repository contradicts itself. The --window-pattern help text in base_train.py describes S as “half context”, and the inline comment on the line above says 2048 -> 768. The arithmetic gives 512, a quarter. The docstring one line further up says “quarter context” and is correct.

The code is right and the comment is stale. In a repository built this carefully, that was genuinely the only inconsistency I found worth reporting — and it is a comment, not behaviour. Trust the arithmetic.

Bar width is the attention window, so the ratio is the whole story. Eighteen of twenty-four layers never see more than 512 tokens; the six gold bars carry every long-range interaction in the model.

Why it should be fast

Everything above is design. This section is what I actually came for: the specific machinery that should keep expensive GPUs busy.

There are five mechanisms, and they are worth naming up front rather than scattering:

1. Fixed shapes and honest batch math — so the compiler can specialize and the global batch never drifts.
2. Attention — FA3 on Hopper, and nothing pretending to be FA3 elsewhere.
3. Precision — FP8 where it is safe, and an explicit refusal where it is not.
4. Communication — the optimizer owns synchronization instead of the model wrapper.
5. The data feeder — dense, fixed-shape batches with no padding at all.

That order runs from inside the model outward: shapes, then kernels, then numerics, then across GPUs, then the pipe feeding it all.

Fixed shapes and honest batch math

The compile call is one line and it sets the tone:

model = torch.compile(model, dynamic=False)

dynamic=False promises the compiler these shapes will not change, so it can specialize hard rather than emit code that handles anything. That is only a safe promise if the rest of the system genuinely never changes shape — which is why the dataloader later goes to such lengths to produce exactly-full rows.

The batch arithmetic is equally direct:

tokens_per_fwdbwd = args.device_batch_size * args.max_seq_len
world_tokens_per_fwdbwd = tokens_per_fwdbwd * ddp_world_size
assert total_batch_size % world_tokens_per_fwdbwd == 0

That assert does more work than it looks. total_batch_size is a property of the experiment — it comes out of the depth cascade and shapes the learning dynamics. device_batch_size is a property of the machine: however much fits in memory. Forcing the two to reconcile through gradient accumulation is what stops the effective batch quietly becoming whatever the hardware allowed.

So GPU count becomes an execution detail rather than a hyperparameter. Run the same configuration on four GPUs instead of eight and you get twice the accumulation steps and an identical global batch. The model does not know how many GPUs it trained on. (In the same spirit, the script takes manual control of Python’s garbage collector after the first step — the mark of someone who profiled their training loop and found GC pauses in it.)

What I expect to see — On four H100s with device_batch_size=16 and max_seq_len=2048, that is 32,768 tokens per rank per micro-batch and 131,072 across four ranks. To reach a 1,048,576-token global batch the script must choose exactly 8 gradient accumulation steps, and should say so.

Attention: FA3, and only on Hopper

nanochat/flash_attention.py tries to load Flash Attention 3, but only on Hopper-class GPUs, and falls back to PyTorch’s SDPA everywhere else behind an API-compatible shim.

What I like is the honesty of the fallback. It is not sold as equivalent. The training script warns that it is meaningfully less efficient, and warns harder if you combine it with sliding windows — a sliding-window mask through the generic path is expensive exactly where FA3 would be cheap. If you are not on Hopper, the advice is to stop pretending and use --window-pattern=L. Hardware-specific paths are labelled as such rather than hidden behind an abstraction that silently underperforms.

What I expect to see — On H100s the run should announce that FA3 is active. If it announces the SDPA fallback instead, every performance number afterwards is measuring something else.

Precision: what FP8 refuses to touch

FP8 is where I expected to find hand-waving, and did not.

base_train.py exposes --fp8, and on CUDA it walks the model converting eligible nn.Linear modules. nanochat/fp8.py is deliberately small: tensorwise dynamic scaling, quantize, torch._scaled_mm, dequantize. Weights use float8_e4m3fn, gradients float8_e5m2 — trading mantissa bits for range, because gradients need the range more.

The interesting word is eligible. The filter skips layers whose dimensions are not divisible by 16, and skips very small layers entirely. Neither rule is arbitrary: FP8 tensor-core paths have alignment requirements, and for a tiny matrix the quantize/dequantize overhead costs more than the faster multiply saves. dev/LOG.md adds that tensorwise scaling beat rowwise at this scale, and that the filtering was needed for stability. This is not a switch flipped because it was available.

So I counted the modules by hand. A depth-24 model has 24 blocks, each with 6 linear matrices, plus the lm_head: 145 conversions. The layers that should fail the filter are the twelve ve_gate projections, each Linear(12, 12) — 12 is not divisible by 16 — and the single smear_gate, which is Linear(24, 1) and far below any sensible minimum. That is 13 skipped, 158 total.

Every linear layer in the model, drawn at the same size so the proportion is honest. The thirteen the filter refuses are not an oversight: twelve ve_gate projections that fail the divisible-by-16 rule, and one smear_gate far below any workable dimension.

What I expect to see — The run should report converting exactly 145 of 158 linear layers and skipping 13. If the numbers differ, my reading of the filter is wrong.

Communication: the optimizer owns it

This is the part of NanoChat I would most want to show someone learning distributed training.

The model is never wrapped in PyTorch’s DistributedDataParallel. Instead, GPT.setup_optimizer() switches to DistMuonAdamW from nanochat/optim.py when the world size exceeds one, and that optimizer does the synchronization itself.

It is a hybrid: AdamW for embeddings, the unembedding and scalar-ish parameters, Muon for the matrix parameters that make up the bulk of the model. The distributed step has three explicit phases:

launch async reductions
  → wait, compute local updates, launch gathers
  → wait for gathers, copy updated parameters back

Each rank receives one slice of the gradient via reduce_scatter, updates only that slice, then all_gathers to rebuild the full parameter. Optimizer state is sharded across ranks — ZeRO-2 in style — so no rank holds a full copy of the Adam moments. For Muon parameters the code goes further, grouping tensors by shape so many small ones move as a few large ones.

Three goals, all visible in the structure: overlap communication with computation, cut optimizer memory, batch small transfers into fewer large ones. And because it is an explicit phase rather than a hook buried in a wrapper’s backward pass, you can see where the distributed boundary is. Gradients stay rank-local through all eight micro-batches — precisely where a DDP-wrapped model would already be all-reducing.

One step, four ranks, one time axis. The dashed line marks where an ordinary DDP-wrapped model would already have been all-reducing; here the gradients are still rank-local, and communication happens afterwards as an explicit async phase.

Not tested here — I can read the design, but I have no profiler trace. I will not claim communication was actually hidden behind computation, only that the code is structured so it could be.

Feeding the GPUs

Fast kernels are useless if the data pipe stalls, and this is the part of the repo I ended up admiring most.

Training data is ClimbMix parquet shards. speedrun.sh grabs 8 first so tokenizer training can start, then downloads the remaining 170 in the background while tokenizer work proceeds — even the download is pipelined. dev/LOG.md calls the switch from FineWeb-EDU to ClimbMix the single biggest improvement to the speedrun time, 2h46m down to 2h01m, and enough of a gain to move the target from d26 to d24. That is the repository’s measurement, not mine.

The runtime loader in nanochat/dataloader.py uses BOS-aligned best-fit packing. Every row begins with a BOS token. The inner loop is:

1. From the buffered documents, take the largest that fits entirely in the remaining space.
2. Repeat until nothing fits.
3. When nothing fits, crop the shortest buffered document to fill the remainder exactly.

Two details are easy to miss. Row capacity is T + 1 = 2049, not 2048, because inputs and targets are the same row offset by one. And the document chosen for cropping is the shortest available, not an arbitrary one, which minimizes what is discarded at that step.

The result is 100% utilization. No padding, ever. Every token in every batch is a real token contributing to the loss.

Successive states of a single row. Two documents are placed whole, then nothing in the buffer fits the remaining 270 tokens, so the shortest buffered document is cropped to fill it exactly. The red block is what that costs.

Why a third of the tokens get thrown away

dev/LOG.md records the cost: BestFit-Crop achieves 100% utilization with roughly 34.6% crop waste at T=2048. That number bothered me. A third of the corpus discarded seemed like a lot for a scheme described as best-fit.

Then I traced where it has to come from.

A document longer than row_capacity can never be selected by the best-fit search, because that search only considers documents that fit entirely in the remaining space. So the only way such a document ever leaves the buffer is the crop path — and the crop path deliberately picks the shortest buffered document. An over-long document therefore has to wait until it is the shortest thing in the buffer, which for a genuinely long document is close to never.

Over-long documents accumulate. The steady-state buffer becomes far more long-tailed than the raw corpus, and the crop rate climbs toward an equilibrium well above anything you would estimate from the document-length distribution alone.

Once you see that, the number stops looking like inefficiency. The 34.6% is not the packer doing a bad job. It is the price of insisting that every row starts at a real document boundary while T stays fixed — paid deliberately, so that every training row is dense, fixed-shape, and free of padding tokens the model would otherwise have to learn to ignore.

SFT makes the opposite trade. scripts/chat_sft.py uses the same packing, but when no conversation fits it pads the row and masks the padding out of the loss. Web text is abundant and interchangeable, so cropping it is cheap; conversations are scarce and structured, and discarding their tails would not be.

What I expect to see — Because every row is filled exactly and nothing is padded, total tokens trained should be exactly iterations × global batch, with no remainder. For 5,568 iterations at 1,048,576 tokens that is 5,838,471,168 — and the run should report that number precisely, not approximately.

Not tested here — The 34.6% figure is the repository’s measurement on real ClimbMix documents. I reproduced the accumulation trend in a small simulation, but did not instrument the actual loader. The prefetching around the loader is code structure, not a measured overlap.

After pretraining

Base pretraining dominates the cost, but it is not the whole pipeline, and the later stages are what turn a checkpoint into something you can talk to.

speedrun.sh downloads a set of synthetic identity conversations, runs scripts/chat_sft.py, then runs chat evaluation. SFT loads the base checkpoint, inherits most hyperparameters from its metadata rather than making you restate them, and reuses the same optimizer setup. The data mixture is SmolTalk, the identity conversations, MMLU, GSM8K, SimpleSpelling, and SpellingBee.

RL exists — scripts/chat_rl.py is a GRPO/REINFORCE-style loop on GSM8K — but it is deliberately simple, with no separate critic or reference-policy copies, and sits outside the speedrun path.

Success for the speedrun is measured as CORE for the base model and ChatCORE after SFT. The target is GPT-2-grade capability: a modest bar by 2026 standards, and a reasonable one for a run meant to cost less than a nice dinner.

What I expect to see

Reading code and predicting behaviour are different activities, and only the second can be wrong in a way you can check. So here is the list I wrote down before renting anything.

The first five are checkable from the run log alone. The last three need the run to finish.

If most of these hold, the interesting claim is not “NanoChat is fast.” It is that a codebase simple enough to read in an afternoon can be this specific about what it is doing, and be right.

What I am not testing

A list of confirmed predictions is only worth something if the scope is honest.

I am not validating the scaling-law choices. I reproduced the derivation and can confirm the code implements what it claims, but whether D^0.383 is the right exponent, or 12 the right token-to-parameter ratio, would need sweeps I have not run.

I am not attributing performance to individual components. If the run hits high utilization, that is the whole stack — model, attention, precision, optimizer, loader, compiler, hardware. Isolating any one needs ablations, and I have none.

I am not measuring kernel-level behaviour: no profiler traces, so no claims about communication actually overlapping computation. And I am not evaluating tokenizer quality, KV-cache internals, or CORE methodology. Real topics, not this post.

Part 2 rents the GPUs and works down the list.

Appendix A — Model configuration

Source: GPTConfig and build_model_meta.

Appendix B — The --depth derivation

Inputs and reference constants:

Derived values:

Reference-model extrapolation follows muP-style transfer from the measured d12 anchor.

References

• Andrej Karpathy, NanoChat.
• Keller Jordan, modded-nanogpt.
• Kaplan et al., Scaling Laws for Neural Language Models.
• Hoffmann et al., Training Compute-Optimal Large Language Models (Chinchilla).
• Yang et al., Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (muP).
• Power Lines: Scaling Laws for Optimal Batch Size.
• The T_epoch framework.
• Horace He, Making Deep Learning Go Brrrr From First Principles — the source of this post’s title phrasing.