Attn Flow

Particles are probability mass drifting toward the tokens each attention head is reading. Brighter tokens pull harder; the bar under a token is how much attention it receives.

The vocabulary

Token: A chunk of text GPT-2 reads — here, words like India, France, Paris.
Logit: The raw score the model assigns each possible next token before normalizing.
Probability: Logits squashed to 0–1 so they sum to one. The flowing particles stand in for where this mass is being pulled.

The six heads

L7·H0 — Pre-crossover. Broad context pull, no decision yet.
L8·H11 — Retrieval A. The final token reaches back through India to France.
L9·H8 — The override head. Attends to India AND Paris at once — frequency contamination. This is the mechanism.
L9·H0 — Induction. Structural copy: capital→capital, of→of.
L10·H0 — Global context. Almost everything drifts to the first token.
L11·H2 — Previous-token. Clean diagonal streaming, each token to the one before it.

Controls

Cycle heads: ◀ ▶ or h/H · Pause: space · Theme: t · Glyph: g · Speed: +/- · Reset: r. On touch, use the buttons at the bottom.

Read the paper · Frequency Wins →

drop extract_attn JSON to load real data

DROP TO LOAD