Timing: 00:00–02:00 (2 minutes).
Teaching intent: Why force every output to use the same compressed memory?
Recall the final-state bottleneck from H1. Contrast one fixed context with a separate query-dependent context for each output. Attention makes earlier representations addressable; it neither creates unlimited memory nor guarantees correct reasoning. The objective is to calculate and test the retrieval mechanism before assembling a Transformer.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 02:00–04:00 (2 minutes).
Teaching intent: Compare a query with keys, then use normalized weights to combine values.
Introduce the complete attention operation, not just a heatmap. In the first phase define the bottleneck and Q/K/V. Next derive scaling, row-wise softmax, and the numerical context. Then distinguish padding and causal masks and self- versus cross-attention. Finish with implementation parity, a value intervention, and two takeaways. Prerequisites: vectors, matrix multiplication, and the H1 distinction between cell state and exposed hidden representations.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 04:00–06:00 (2 minutes).
Teaching intent: A translation decoder may emphasize different source positions at different steps.
This matrix is hand-designed conceptual alignment, not a measured translation result. Rows represent output queries and columns encoder positions. Source-target word alignment can be soft and need not be one-to-one.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 06:00–08:00 (2 minutes).
Teaching intent: LSTM or GRU encoders can supply states to an attention mechanism.
Bahdanau and colleagues proposed joint alignment and translation in a recurrent system. The 2014 preprint was presented at ICLR 2015. Removing recurrence came later in the Transformer architecture.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 08:00–10:00 (2 minutes).
Teaching intent: The decoder can retrieve from H instead of using only the final state.
All encoder states form an addressable memory. The decoder query selects which states contribute for the current prediction. Access cost grows with memory length, and the states themselves remain learned representations.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 10:00–12:00 (2 minutes).
Teaching intent: The query is a vector used to compare the current need with memory keys.
The picture shows chosen two-dimensional vectors. A natural-language query and an attention query vector are different objects; a learned projection can produce the latter. Avoid claiming every learned coordinate has a named semantic meaning.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 12:00–14:00 (2 minutes).
Teaching intent: Matching uses keys; the retrieved combination uses values.
Keep each key/value row aligned. Values may have a different width from keys. We use the same row ordering throughout the hand calculation.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 14:00–16:00 (2 minutes).
Teaching intent: A score compares one query with one key before normalization.
Dot-product scores can be negative and do not sum to one. Here q=[1,0] gives A and C score 1 and B score 0. Ask students whether a zero score means zero attention weight.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 16:00–18:00 (2 minutes).
Teaching intent: An MLP can compare a decoder query with each encoder key.
This is the additive scoring family associated with Bahdanau attention. Wq and Wk align possibly different widths. Softmax and the weighted value sum follow scoring, just as they do for dot-product attention.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 18:00–20:00 (2 minutes).
Teaching intent: Divide by √dₖ so high-dimensional dot products are less likely to saturate softmax.
Under independent zero-mean unit-variance coordinates the dot-product variance grows with d_k. Scaling reduces that variance. It is not cosine normalization and does not eliminate every saturation issue.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 20:00–22:00 (2 minutes).
Teaching intent: Normalize separately for each query over its available keys.
Subtracting the maximum score stabilizes exponentiation without changing the result. Every unmasked row sums to one. A finite low score still gets positive weight; masking introduces exact zeros.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 22:00–24:00 (2 minutes).
Teaching intent: Each value contributes according to its attention weight.
Weighted averaging combines values rather than selecting a single argmax key. The context width is d_v. With nonnegative normalized weights, the context lies in the convex hull of values before later projections and residual additions.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 24:00–26:00 (2 minutes).
Teaching intent: Rows are query destinations; columns are key sources.
There can be different numbers of queries and keys in cross-attention. Multiplication by V of shape Nk by d_v produces Nq by d_v. Students label axes before doing arithmetic.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 26:00–28:00 (2 minutes).
Teaching intent: The values are deliberately small so all intermediates can be checked.
Ask students to compute all three raw dot products on paper. These are hand-designed vectors, not a learned language example. Use the table from slide 7 to avoid key/value mix-ups.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 28:00–30:00 (2 minutes).
Teaching intent: A zero dot product still receives a positive normalized weight.
Students calculate sqrt(2), exponentials, and row normalization. A and C tie. B receives approximately .1978 rather than zero because exp(0)=1.
Ask: What are the three attention weights?
Expected answer: Approximately [0.4011, 0.1978, 0.4011], with sum 1.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 30:00–32:00 (2 minutes).
Teaching intent: Use the same weights for every value coordinate.
The first coordinate is .4011*10+.1978*0+.4011*10. The second is .4011*0+.1978*10+.4011*10. Bars illustrate weighted contribution magnitudes, not prediction errors. Verify in H2 Step 5.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 32:00–34:00 (2 minutes).
Teaching intent: Attention can route the same memory differently for each request.
Use the query selector and temperature slider. Ask students to predict the rank before clicking. Temperature 1 reproduces the hand example. Values are fixed; changed contexts come only from changed routing.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 34:00–36:00 (2 minutes).
Teaching intent: Q, K, and V come from learned projections of the same representations.
The three projections have different roles even when their input is the same. Self-attention is not limited to language. A token may represent a word piece, image patch, or sensor-time feature vector.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 36:00–38:00 (2 minutes).
Teaching intent: Queries come from one sequence; keys and values come from another.
The original Transformer decoder uses encoder-decoder cross-attention. A decoder-only model normally uses masked self-attention and has no separate source encoder. This distinction is essential before H3 architecture comparison.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 38:00–40:00 (2 minutes).
Teaching intent: Before softmax, set scores for key positions j>i to −∞.
Diagonal entries remain allowed because input token i predicts token i+1. A causal mask blocks access to later input positions, not the current one. Softmax must follow masking.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 40:00–42:00 (2 minutes).
Teaching intent: Padding and causality enforce different constraints.
Padding marks filler positions that do not correspond to observations. A causal mask depends on relative positions. Their intersection defines available keys. Padded query outputs should be excluded from the training loss.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 42:00–44:00 (2 minutes).
Teaching intent: Softmax requires at least one permitted source per query.
Do not silently convert NaN to a claim. Avoid all-masked rows or define a deliberate special-case behavior. Boolean mask conventions differ across APIs; inspect the actual API used.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 44:00–46:00 (2 minutes).
Teaching intent: A query can access a distant key within one dense attention layer.
This can ease access to long-range information. It does not imply perfect reasoning, unlimited context, or guaranteed retrieval. A fixed-size context window and representation quality still constrain performance.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 46:00–48:00 (2 minutes).
Teaching intent: Doubling context length quadruples score entries at fixed head count.
Count pairs rather than asserting all computation is quadratic: dense attention score interactions are O(T^2*d_k), while feed-forward cost also depends on width. Fused exact kernels can avoid materializing the whole score matrix, without changing dense pair count.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 48:00–50:00 (2 minutes).
Teaching intent: Verify your hand calculation before exploring masks.
Use H2 Steps 1 through 5. Students show score and weight rows to a partner. The notebook is independent and requires no trained H1 checkpoint.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 50:00–52:00 (2 minutes).
Teaching intent: Every permitted row sums to one; every future weight is zero.
Run Steps 7 and 8. The displayed matrix is illustrative; actual notebook values depend on the chosen vectors. Students explain both assertions and identify source/destination axes.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 52:00–54:00 (2 minutes).
Teaching intent: NumPy and PyTorch should agree for the same inputs, scale, and mask.
Run Step 9. There is no dropout in this comparison. Passing parity verifies the selected numerical operation and does not establish model quality. In SDPA a Boolean True means allowed, unlike certain other PyTorch attention masks.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 54:00–56:00 (2 minutes).
Teaching intent: Changing values can alter the output while attention weights remain fixed.
Run Step 10. Explain that downstream layers and residual paths also affect predictions. An attention heatmap describes routing in a computation; it is not sufficient evidence for a causal explanation of the output.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 56:00–58:00 (2 minutes).
Teaching intent: State the sequence: compare, normalize, retrieve.
Pairs explain the full operation without reading the formula. One partner changes the query; the other predicts the new ranking. Require the saved attention evidence JSON and one limitation.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]
Timing: 58:00–60:00 (2 minutes).
Teaching intent: Attention retrieves a query-dependent mixture of values. Masks determine which information is available.
Close with two ideas: attention is query-dependent weighted retrieval, and masking defines the allowed information. Ask a student to identify the query and key axes and explain why a zero dot product is not a zero softmax weight. Remind students that attention weights alone are not causal explanations. The next hour places this mechanism inside a complete decoder block.
Visual: Original editable SVG or live calculation, designed for this slide. All non-run numeric diagrams are illustrative teaching calculations.
[Sources]