Trace the complete computation, then test whether future tokens can affect earlier predictions.

Goal: explain a block and preserve a causal training experiment.

Unmasked attention alone is permutation-equivariant when positions are absent.

Attention(PX) = P Attention(X)

The teaching model adds learned position vectors; the original Transformer used sinusoidal encodings.

zₜ = token_embedding(idₜ) + position_embedding(t)

Each head routes information in its own learned query/key/value subspace.

D = H × dhead

Head outputs are joined and mixed by a learned output projection.

MultiHead = Concat(head₁,…,headH) WO

The same nonlinear MLP is applied independently at each position.

FFN(x) = W₂ GELU(W₁x + b₁) + b₂

A sublayer adds a learned update to its input representation.

y = x + Sublayer(x)

Our teaching decoder normalizes before each sublayer.

x ← x + Attention(LN(x))

Encoder self-attention can access all valid input positions.

Masked self-attention supports next-token prediction from a prefix.

Decoder queries can retrieve from the separate encoder outputs.

Shift targets one step ahead of inputs.

input: [0,1,2,3] target: [1,2,3,4]

The model should assign high probability to observed next-token targets.

L = −mean log p(yₜ | x≤ₜ)

The causal mask permits computing many training losses together.

One block, width 32, four heads, and eight symbols keep the experiment small.

Separate streams use different noise seeds but share the same synthetic process.

Every head produces a query-by-key grid for each example.

scores: (B, H, T, T)

The output has one vocabulary-logit vector per input position.

IDs: (B,T) → logits: (B,T,V)

Estimate the next state from the current state using transition counts.

P(next | current) = smoothed count / row total

Keep the checkpoint with lowest validation negative log-likelihood.

Rows are receiving positions; columns are source positions.

Change tokens after position 7; logits at positions 0–7 should remain identical.

Δearlier = max |old_logits − new_logits|

Permute inputs to unmasked attention without positional information.

Autoregressive decoding grows the prefix sequentially.

Dense attention pair counts grow as T² at fixed head count.

T=16: 256 pairs T=32: 1024 pairs

Streaming state, context length, data size, and compute budget all matter.

An agent also needs tools, state management, permissions, and a stopping rule.

Record data identity, configuration, baseline, checkpoint, and observed tests.