The engineering notebook

The maths is checked against a second method, not against itself

The network on the front page computes its own gradients. A backward pass that is subtly wrong still trains, still produces a loss curve that falls, and still writes plausible text, which makes it one of the easier things in software to be confidently wrong about. So the analytic gradient is never trusted on its own.

Every parameter is checked against a central finite difference: nudge one weight up, nudge it down, measure how the loss actually moved, and compare that with what the backward pass claimed. Every parameter, not a sample — the embedding table included, which is where an indexing mistake hides best. The two methods share no code, which is the entire point of using two.

That is still not enough by itself, because a suite that passes the first time it runs is a suite nobody has tested. So there is a second harness whose job is to break the maths on purpose: fifteen plausible mistakes, each one written into the model file and then run. The gradient passed through units that ReLU switched off. The sign flipped at the logits. Temperature applied after the softmax instead of before. Top-k keeping the least likely characters. The vocabulary left unsorted, so the indices depend on the order the text arrived in. Every one of those has to turn the suite red, and the harness fails if any of them survives. A mutation that survives is not a mutation to delete; it is a test that is missing.

The attention you see is measured, not drawn

The attention heatmap on this page used to be illustrative, and said so in a caption, which is the honest version of a bad answer. It is now a real two-layer, two-head transformer with 65,184 parameters, trained offline on the full tinyshakespeare corpus with a sha256-pinned download, and shipped as weights the page runs a forward pass over.

Training it elsewhere and running it here creates exactly the problem you would expect: two implementations of the same architecture, in two languages, drifting apart. So the training script writes out a golden file — a fixed input with the logits and attention vectors PyTorch produced for it — and the browser implementation replays that input and is compared against it on every run. The current agreement is 2.91e-6 on the logits and 7.99e-7 on the attention weights, which is single-precision noise and not a second opinion.

The first attempt at this model trained on a 17KB corpus and memorised it: training loss 0.60 against a validation loss of 5.13. The attention patterns it produced looked convincing and meant nothing. Using the full corpus is what makes the picture on the page a picture of something.

Every number the page says about itself is derived

A page that states its own model has 1,075 parameters and then grows the model has published a false sentence, and nothing about the way it fails will tell you. So the counts are not typed. The parameter count is computed by walking the real weight arrays, and the claims suite counts them again independently and compares the result against every rendered sentence that mentions a number — sampled in more than one place, because the same claim is worded differently in the running status and the idle placeholder, and the two never appear on screen together.

The same rule covers the corpus sizes, the vocabulary size, the duration the hero card promises and the paper years in the reference list. Thirty papers are archived beside the page with their lengths and hashes, and the citation suite checks which ones are cited and with what year.

Where this page was wrong first

The hero card overstated the model by 3.3 times. It said "3,500 random numbers" in three places, for a network with 1,075 parameters. The prose had been written for a larger configuration and was never revisited when the hero was slimmed down for speed. The fix was structural rather than editorial: the count is read from the weights now, and only one literal survives.

And promised sixty seconds for a ten-second run. Same class of drift, opposite direction, and the research says the inflated number was costing clicks as well as being false: a time estimate near a call to action lowers perceived effort, so the wrong one suppressed the very thing it was there to encourage.

A gate that agreed with the code and disagreed with the visitor. Two elements carried the hidden attribute and were visible anyway, because an author display rule outranks it. A first-time visitor was being offered a button to replay a demo they had never seen. The check that was supposed to catch this asserted element.hidden — the DOM property, which read true the whole time. It asserts computed display now, and a second check generalises it across the site.

The suite ran green against the wrong website. The pre-deploy script reused whatever server was already listening on its port, and a leftover one from the repository this project grew out of answered. Two hundred and eight checks passed against a different index.html, and were reported as a pass. The script starts its own server on a free port now.

Two deploys in the same minute served half of each. The cache-busting stamp had minute granularity, and a few assets are excluded from content-hashed filenames because their URLs have to stay stable. Another session deployed in the same minute, so the edge already had that URL cached from their build, and the site served a new script against an older model file — a real functional break, not a cosmetic one. A zone-wide purge did not clear it; only purging the exact URL did. There is a check now that compares every excluded asset on the live site against what was just built.

The tap that did nothing. The badge notification invited you to tap it to see your collection, and the handler tested whether the click target carried the hidden-section class — true only of the section itself, never of anything inside it. So the tap fell through to a scroll on an element with a hidden ancestor, which silently does nothing. It was worst for the first badge anyone earns, which always arrives before that section opens: a new visitor's very first tap was guaranteed dead.

A ten-second run that froze for six of them. The hero demo was measured, not watched: nineteen freezes totalling 5.9 seconds of a 10.0-second run. Chained timers, a full rebuild of the generated text on every token, and two axes rescaling on every frame. It runs against the display clock now, and the same run measures zero stalls.

Prose set on a 120-character line, and a measurement that only saw a third of the page. Found in August 2026 by measuring rather than looking: body paragraphs running 105 to 128 characters, where the readable band is 45 to 75. The first pass reported six such runs. It was wrong, and wrong in the way that matters: it opened the locked sections by removing a class, the sections stayed hidden anyway, and four of the ten — embeddings, training, generation and scale up, which are most of this page — were never measured at all. Unlocking them the way a visitor does grew the page, at a 1280-pixel viewport, from about 28,000 pixels to about 81,000, and the count from six to sixty-three. Fifty-four of those were inside bordered boxes, which were then capped to hug their text.

And the cap was the wrong answer. Reported in September 2026: the capped boxes stopped 50 to 135 pixels short of the panels and figures stacked beside them, and one box squeezed a diagram into two thin columns. They fill their card again. The line length that cost was bought back with type size instead, at 17 pixels with more padding and leading, which measures a median of 79 characters: the same as this page's own paragraphs, and inside the 80 WCAG asks for. Two things were wrong with the old check as well. It estimated characters by dividing the box width by an average glyph, which read about 7 per cent low and hid lines of 100. And it had one number for every layout, when the readable length depends on how many columns of text sit side by side: 40 to 50 per column for paragraphs, about 25 for short copy in a tile, and below that the columns should stack rather than squeeze. Both are counted from real rendered lines now, and two grids that set paragraphs three across at 22 to 26 characters became rows.

What is not here

A server. Nothing on this page calls one. The model trains in the tab, in JavaScript, on the visitor's own machine, and nothing here sends the weights anywhere — the only way they move is the export button, which writes a file when you press it. That is a constraint as much as a feature: it is why the model is small enough to be honest about, and why the page can say that what you are watching is your model rather than a recording.

A large model, unless you ask for one. The Scale Up section runs a 135-million-parameter model in the browser, and it is a deliberate download rather than something the page fetches for you.

Benchmarks and vendor lore. The page explains mechanisms and cites papers. It does not rank products, and where a question is genuinely contested in the literature — whether capabilities emerge sharply with scale, for instance — it says it is contested rather than picking the more quotable side.

A light mode. Not an omission. The studio's position is that these are instruments and instruments are dark, and the page now declares that to the browser so its own controls match rather than rendering as white slabs on a black ground.

Sources

The full reference list, with what each paper is cited for, is on the front page. These are the ones this notebook leans on directly.

• Y. Bengio, R. Ducharme, P. Vincent and C. Jauvin, A Neural Probabilistic Language Model, JMLR 3, 2003.
• D. Rumelhart, G. Hinton and R. Williams, Learning representations by back-propagating errors, Nature 323, 1986.
• A. Vaswani and others, Attention Is All You Need, 2017.
• X. Glorot and Y. Bengio, Understanding the difficulty of training deep feedforward neural networks, AISTATS, 2010.
• G. Hinton, O. Vinyals and J. Dean, Distilling the Knowledge in a Neural Network, 2015.
• A. Holtzman and others, The Curious Case of Neural Text Degeneration, 2019.
• R. Sennrich, B. Haddow and A. Birch, Neural Machine Translation of Rare Words with Subword Units, 2015.
• R. Pascanu, T. Mikolov and Y. Bengio, On the difficulty of training Recurrent Neural Networks, 2012.