A GPT-style decoder, one idea at a time: tokens, embeddings, residual stream, attention heads, MLP, then a next-word distribution. The live net is tiny (32-d, 2 blocks) and uses whole words so every attention cell stays readable. Clinical snippets are the teaching example, not the definition.
Section 1
Wikipedia: Transformer, Attention (machine learning). Course: CNN explainer.
A transformer reads a whole sequence at once and lets every position talk to every other position through self-attention. It was introduced in 2017 (Attention is All You Need) and now powers models such as GPT, Llama, and Gemini.
RNNs squeeze the past into one hidden state. CNNs, like the CNN explainer, look at a local window. A transformer can jump from the last word back to a name mentioned twenty tokens earlier, in one step.
This lab is a decoder-only GPT: given the tokens so far, predict the next one. That single job, scaled up, is how large language models write. Guided steps below scroll you through the page; nothing is hidden.
Step 1 / 8
Section 2
Wikipedia: Tokenization.
The live net maps each whitespace word to an ID in a table of 87 rows. That is deliberately not how GPT-2 works — whole words keep the attention matrix readable. Click a chip to set the query token used in the attention lab later.
Unknown words become UNK. Do not paste real notes. This teaching net has … parameters.
Section 3
Wikipedia: Byte-pair encoding.
Type here; the split updates immediately. This playground is independent of the 87-word live net. GPT-2’s vocabulary has 50,257 pieces. Do not paste real patient text.
Two tokenizers, on purpose: the diagram uses whole words so you can read every attention cell. This box shows how a production GPT-2 tokenizer would shatter the same kind of string — including lab symbols and emoji bytes.
Section 4
Wikipedia: Word embedding.
Each token ID becomes a vector of length \(d_{\text{model}} = 32\) (GPT-2 small uses 768). A second table stores a vector for each position \(0 \ldots 19\). The input to block 1 is the sum \(x_t = E_{\text{token}}[w_t] + E_{\text{position}}[t]\). GPT-2 learns those position vectors rather than using the 2017 sine/cosine table.
This net and GPT-2 add learned position embeddings. Many newer LLMs use RoPE: they rotate Q/K so relative distance is baked into the dot product instead of a lookup table.
Section 5
Course: CNN explainer (local windows vs global attention).
Each row is one token. Click a token to highlight its lane. The Attention column names the key this query currently weights most — open the lab below for the full matrix. Residual arcs skip around attention: they are bypasses, not extra lectures.
Section 6
Wikipedia: Self-attention.
After layer-norm, three linear maps make query (what this token asks), key (what others advertise), and value (what actually gets mixed in). Per head the width is 8 numbers. Click a token above to change which row you inspect.
Prefill reads the whole prompt in parallel. During generation, past keys and values are cached. This page recomputes the tiny net each time so you can see every cell.
Section 7
Wikipedia: Attention, Softmax.
\(Q = X W_Q,\quad K = X W_K,\quad V = X W_V\), then \(\mathrm{Attention}(Q,K,V) = \mathrm{softmax}((QK^\top)/\sqrt{d_{\text{head}}} + M) V\). The mask \(M\) is \(0\) on and below the diagonal and \(-\infty\) above it: token \(t\) may only look at keys \(\le t\).
A bright cell means this query used that key. It is not a causal explanation of the prediction. See interpretability.
Section 8
Four 8-d heads run in parallel, then concatenate to 32-d and mix with \(W_O\). Switch heads in the lab: the heatmap should change; the residual width stays 32. Heads are not labelled clinical concepts. GPT-2 small stacks this recipe 12 times; this lab stacks it twice.
Standard multi-head attention gives every head its own K and V. Multi-query and grouped-query attention share keys/values so the KV cache stays small — important at long clinical context, not used in this toy.
Section 9
Wikipedia: Residual neural network.
Attention routes between tokens. The MLP (32→128 GELU→32) only rewrites the current token’s features, then adds them back: \(x \leftarrow x + F(x)\). Residual connections keep a copy of the old stream so gradients still reach the embeddings.
GPT-2 uses a GELU MLP. Many newer models use SwiGLU or similar gated variants. Same job (rewrite this token), different nonlinearity.
Section 10
Wikipedia: Softmax, Autoregressive model. Course: calibration & thresholds.
Only the last position is used to pick the next word. Temperature \(T\) divides logits before softmax; top-k keeps \(k\) survivors. Neither knob adds new evidence. Click a bar to mark that guess as coming from the last lane.
Softmax over the closed teaching vocabulary from the last token’s final vector.
This net was trained for 18 epochs on synthetic notes; final train loss 0.51. Generation is still one token at a time. Training is not.
| Position | Sees (input) | Must predict (target) | At generation time |
|---|---|---|---|
| 0 | The | patient | Not used to write the next word of an already-started note |
| 1 | The patient | has | Same: a training signal only |
| 2 | The patient has | fever | This last lane is what sampling reads |
Section 11 · coda
Course: dense EHR explainer, interpretability, calibration.
The architecture does not change when the text is a discharge summary. The data does. Notes are long, copied forward, full of negation (“no fever”), and stuffed with abbreviations. Tabular EHR models see one row of numbers. A transformer sees order: “penicillin allergy” before “prescribed”.
The bars sum to 1 over a toy vocabulary. A 0.99 “insulin” score is not a 99% chance that insulin is indicated. See calibration & thresholds.
A bright cell means “this query used that key.” It does not mean “creatinine caused the prediction.” See interpretability.
A 3×3 convolution looks at a neighbourhood of pixels. Attention can jump from the last word to an allergy mentioned 10 tokens earlier. See the CNN explainer.
This net memorised a few synthetic templates. Real clinical language models need huge corpora, de-identification, and external validation.
Positions: learned embedding table (this lab, GPT-2) vs RoPE in most current LLMs.
Norm: LayerNorm vs RMSNorm.
MLP: GELU expansion vs gated SwiGLU-style MLPs.
Attention: every head has its own K/V (MHA) vs grouped- or multi-query attention to shrink the KV cache.
Inference: this page reruns the whole net; production servers cache keys and values after the prompt prefill.
| This lab | GPT-2 small | Clinical language models | |
|---|---|---|---|
| Objective | Next word on synthetic notes | Next token on web text | Often masked LM (BERT-style) on notes, or decoder LMs on de-identified corpora |
| Parameters | ~32 thousand | 124 million | Typically tens of millions to billions (Med-BERT, GatorTron-class, local LLMs) |
| Vocabulary | 87 whole words | 50,257 BPE pieces | WordPiece / BPE plus institution-specific artefacts |
| Context | 20 tokens | 1,024 in GPT-2 | Notes can run to thousands of tokens; truncation is a silent bias |
| Bidirectional? | No (causal mask) | No | Encoders (BERT / Med-BERT) see both sides; generators usually cannot |
| Safe to deploy? | No | No | Only after task-specific validation, calibration, and workflow design |
“Denies chest pain”, “rule out pneumonia”, “cannot exclude PE”. If attention only sees the noun, the phenotype flips. Try the fever / no-fever pair.
Yesterday’s “on metformin” stays in the note after AKI. A next-token model will keep writing the old story unless the new creatinine actually outranks the copied sentence.
NKDA, SOB, PT (prothrombin? physical therapy? patient?). A closed vocab makes the problem obvious; a 50k BPE vocab hides it inside pieces.
This page never leaves your browser and never saw a real note. Hospital models trained on notes can regurgitate identifiers. That is a privacy issue, not an architecture footnote.
If every “rising creatinine” sentence in training ends with insulin, the model has not learned nephrology. It has learned a template. That is this entire demo, on purpose.
High attention on “injury” may be real routing, or a positional habit. Do not put heatmaps in a model card as if they were a causal DAG. See causal inference.