Timing: 00:00–02:00 (2 minutes).
Teaching intent: Attention mixes positions. A complete decoder must also represent order and predict the next token.
State that this hour implements a small pre-normalized decoder-only model, not a literal replica of the 2017 encoder-decoder. Introduce the whole path: token and position vectors, a masked-attention residual update, a tokenwise feed-forward residual update, final normalization, vocabulary logits, and next-token prediction. This is a language-model mechanism exercise on synthetic symbols, not an industrial LLM or an autonomous agent.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 02:00–04:00 (2 minutes).
Teaching intent: Trace the complete computation, then test whether future tokens can affect earlier predictions.
Tokens are discrete units: token IDs index learned vectors; numerical ID distance is not semantic distance. Use the following position slide to connect token embeddings with order. Explain the block, train a small model with a bigram baseline, inspect real attention, and change future tokens to test causality. The evidence is an actual checkpoint, data/configuration record, test NLL, and observed causal-test result.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 04:00–06:00 (2 minutes).
Teaching intent: Unmasked attention alone is permutation-equivariant when positions are absent.
State the precise equivariance claim for unmasked self-attention with shared projections. A causal mask introduces order constraints, so the claim is not directly the same for a masked decoder. The notebook tests the unmasked operation separately.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 06:00–08:00 (2 minutes).
Teaching intent: The teaching model adds learned position vectors; the original Transformer used sinusoidal encodings.
Learned absolute positions are bounded by the allocated context table. Sinusoidal and rotary approaches are alternatives with different behavior. Do not claim every modern LLM simply adds the same sinusoidal encoding.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 08:00–10:00 (2 minutes).
Teaching intent: Each head routes information in its own learned query/key/value subspace.
Different heads can learn different patterns but are not assigned fixed human roles. The notebook uses model width 32 and four heads of width 8. Every head has its own projected representation.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 10:00–12:00 (2 minutes).
Teaching intent: Head outputs are joined and mixed by a learned output projection.
With fixed total width D, increasing head count reduces per-head width; it does not automatically multiply total width. Emphasize that projection weights are learned and attention weights are input-dependent.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 12:00–14:00 (2 minutes).
Teaching intent: The same nonlinear MLP is applied independently at each position.
Attention mixes information between positions. The tokenwise MLP transforms features locally and adds nonlinearity. Our notebook uses GELU; the original 2017 formulation used ReLU.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 14:00–16:00 (2 minutes).
Teaching intent: A sublayer adds a learned update to its input representation.
Shapes must match for addition. This provides an identity path and helps optimization; it does not mean the sublayer is optional after training. The diagram highlights the bypass separately from the update.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 16:00–18:00 (2 minutes).
Teaching intent: Our teaching decoder normalizes before each sublayer.
This is pre-normalization. The original Transformer used post-normalization and a full encoder-decoder. Label the variant explicitly so students do not treat a modern educational block as a verbatim copy of the 2017 figure.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 18:00–20:00 (2 minutes).
Teaching intent: Encoder self-attention can access all valid input positions.
This illustrative matrix shows full-context access. A padding mask may still apply. Full access is suitable when the entire observed input is available, but future unknown observations cannot be treated as valid inputs in forecasting.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 20:00–22:00 (2 minutes).
Teaching intent: Masked self-attention supports next-token prediction from a prefix.
Our tiny model has a causal self-attention block and no cross-attention sublayer. Input position i predicts the next token i+1, so position i itself is visible. Explain diagonal access.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 22:00–24:00 (2 minutes).
Teaching intent: Decoder queries can retrieve from the separate encoder outputs.
Vaswani et al. introduced an encoder-decoder translation model. Decoder-only is a later architecture family commonly used for generative language modeling. Architecture choice follows the task, not a single mandatory recipe.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 24:00–26:00 (2 minutes).
Teaching intent: Shift targets one step ahead of inputs.
The final target lies beyond the input sequence. The mask prevents a position from seeing its own next-token answer elsewhere in the input. Students verify the shifted-target assertion in notebook Step 2.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 26:00–28:00 (2 minutes).
Teaching intent: The model should assign high probability to observed next-token targets.
These bars are mathematical examples: uniform eight-way probability, probability .5, and probability .8, giving negative log probabilities. They are not trained-model results. Nats/token uses the natural logarithm.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 28:00–30:00 (2 minutes).
Teaching intent: The causal mask permits computing many training losses together.
Parallel computation during teacher-forced training does not make arbitrary autoregressive generation parallel. During generation each next input depends on the token just produced. Cached states may reduce repeated work but do not remove that dependency.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 30:00–32:00 (2 minutes).
Teaching intent: One block, width 32, four heads, and eight symbols keep the experiment small.
Launch H3 notebook. Setup does not depend on earlier saved files. All data are generated locally. The objective is mechanism and testing rather than producing fluent natural language.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 32:00–34:00 (2 minutes).
Teaching intent: Separate streams use different noise seeds but share the same synthetic process.
These streams are separate generated realizations of the same motif. They are not disjoint natural-language documents or evidence of transfer to a new engineering domain. State the scope before looking at loss values.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 34:00–36:00 (2 minutes).
Teaching intent: Every head produces a query-by-key grid for each example.
Read the split and transpose operations in CausalAttention. Explain why the final softmax dimension is keys. A transpose error can produce a plausible shape with the wrong meaning.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 36:00–38:00 (2 minutes).
Teaching intent: The output has one vocabulary-logit vector per input position.
Run the shape assertion. Logits are unrestricted real scores, not probabilities. Cross-entropy accepts logits directly, so a separate softmax before nn.CrossEntropyLoss is unnecessary.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 38:00–40:00 (2 minutes).
Teaching intent: Estimate the next state from the current state using transition counts.
The slide matrix is illustrative; run the actual training counts in Step 5. Add-one smoothing avoids zero test probability. The motif can have ambiguous next states, so longer context may outperform a bigram.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 40:00–42:00 (2 minutes).
Teaching intent: Keep the checkpoint with lowest validation negative log-likelihood.
Run Step 6. AdamW updates parameters; clipping controls gradient magnitude. Students should record the actual training duration and loss curve rather than copy a predetermined score.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Measured figure: h3-training-loss.png; recorded CPU notebook execution, 19 October 2026. Source arrays and checks: validation/result-figure-provenance.json.
Timing: 42:00–44:00 (2 minutes).
Teaching intent: Rows are receiving positions; columns are source positions.
Run Step 7. The plot is a real output of the students’ training run. Do not interpret a zero upper triangle as a learned dependency: masking enforces it. Compare multiple heads if time permits.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Measured figure: h3-trained-attention.png; recorded CPU notebook execution, 19 October 2026. Source arrays and checks: validation/result-figure-provenance.json.
Timing: 44:00–46:00 (2 minutes).
Teaching intent: Change tokens after position 7; logits at positions 0–7 should remain identical.
Run the causality assertion in Step 8. This is a behavioral check of the computation, independent of accuracy. Use model.eval and no_grad to remove irrelevant stochastic effects. The test covers the supplied intervention, not every hypothetical failure.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 46:00–48:00 (2 minutes).
Teaching intent: Permute inputs to unmasked attention without positional information.
Run the isolated equivariance assertion. Then remove positions at inference in the trained model and record NLL. An inference intervention is not a fair comparison to a separately trained no-position model; make that distinction explicit.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 48:00–50:00 (2 minutes).
Teaching intent: Autoregressive decoding grows the prefix sequentially.
Run Step 10. The notebook uses argmax for a deterministic demonstration. Temperature sampling would change diversity. Cropped windows reuse learned absolute position indices, which is a limitation of the simple implementation.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 50:00–52:00 (2 minutes).
Teaching intent: Dense attention pair counts grow as T² at fixed head count.
Distinguish pair-count scaling, memory storage, and elapsed time. Fused attention implementations can reduce intermediate memory use. Do not claim our tiny CPU timing predicts an industrial LLM’s latency.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 52:00–54:00 (2 minutes).
Teaching intent: Streaming state, context length, data size, and compute budget all matter.
Neither architecture wins universally. LSTM is useful when compact streaming state is appropriate; dense Transformers provide flexible context mixing with context-dependent costs. Baseline evidence decides the engineering choice.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 54:00–56:00 (2 minutes).
Teaching intent: An agent also needs tools, state management, permissions, and a stopping rule.
Connect this week to the later agent units. No tool execution or persistent planning emerges merely because a Transformer can generate tokens. The tiny notebook does not call external services or perform consequential actions.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 56:00–58:00 (2 minutes).
Teaching intent: Record data identity, configuration, baseline, checkpoint, and observed tests.
Students save the evidence JSON and model state. Ask them to name one controlled change and one unsupported broader claim. Explain which artifacts disappear when a Colab session ends.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 58:00–60:00 (2 minutes).
Teaching intent: A causal decoder combines attention with position and feature transformations. Its predictions still need evidence.
Finish with two takeaways: attention is one component of a complete causal architecture, and a working implementation must be evaluated rather than assumed correct. Have a student trace inputs to logits and explain why changing future tokens must preserve earlier logits. Save data identity, seeds, versions, baseline results, checkpoint, and limitations. Retrieval weights are not causal explanations; a decoder alone supplies neither tools nor permission controls. This is classroom practice, not a new graded weekly assignment.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]