Why force every output to use the same compressed memory?
Machine Learning in Engineering Science Fall 2026 · Week 07 · Hour 2 Chi-Hua Jonny Yu · NCKU / LAiMM
W07 · H2 · 15:00–16:00 · 19 October 2026
Attention: selective retrieval
Machine Learning in Engineering Science · 19 Oct 202600:00–02:00 · 01/30
W07 · H2 · Machine Learning in Engineering Science
This hour: learn to retrieve
Compare a query with keys, then use normalized weights to combine values.
Goal: compute a context vector and explain a causal mask.
Machine Learning in Engineering Science · 19 Oct 202602:00–04:00 · 02/30
W07 · H2 · Machine Learning in Engineering Science
Alignment is output-dependent
A translation decoder may emphasize different source positions at different steps.
Machine Learning in Engineering Science · 19 Oct 202604:00–06:00 · 03/30
W07 · H2 · Machine Learning in Engineering Science
Attention predates the Transformer
LSTM or GRU encoders can supply states to an attention mechanism.
2014/15: recurrent attention 2017: Transformer
Machine Learning in Engineering Science · 19 Oct 202606:00–08:00 · 04/30
W07 · H2 · Machine Learning in Engineering Science
Keep the full set of encoder states
The decoder can retrieve from H instead of using only the final state.
H = [h₁; h₂; …; hₙ]
Machine Learning in Engineering Science · 19 Oct 202608:00–10:00 · 05/30
W07 · H2 · Machine Learning in Engineering Science
A query specifies what to look for
The query is a vector used to compare the current need with memory keys.
q ∈ ℝᵈᵏ
Machine Learning in Engineering Science · 19 Oct 202610:00–12:00 · 06/30
W07 · H2 · Machine Learning in Engineering Science
Keys address; values carry content
Matching uses keys; the retrieved combination uses values.
K ∈ ℝᴺˣᵈᵏ V ∈ ℝᴺˣᵈᵛ
Machine Learning in Engineering Science · 19 Oct 202612:00–14:00 · 07/30
W07 · H2 · Machine Learning in Engineering Science
A score is a compatibility function
A score compares one query with one key before normalization.
dot score: eⱼ = q · kⱼ
Machine Learning in Engineering Science · 19 Oct 202614:00–16:00 · 08/30
W07 · H2 · Machine Learning in Engineering Science
Additive attention learns a scoring network
An MLP can compare a decoder query with each encoder key.
eᵢⱼ = vₐᵀ tanh(Wq qᵢ + Wk kⱼ + b)
Machine Learning in Engineering Science · 19 Oct 202616:00–18:00 · 09/30
W07 · H2 · Machine Learning in Engineering Science
Scaled dot products control score magnitude
Divide by √dₖ so high-dimensional dot products are less likely to saturate softmax.
S = QKᵀ / √dₖ
Machine Learning in Engineering Science · 19 Oct 202618:00–20:00 · 10/30
W07 · H2 · Machine Learning in Engineering Science
Softmax allocates a unit total weight
Normalize separately for each query over its available keys.
aᵢⱼ = exp(sᵢⱼ) / Σₘ exp(sᵢₘ)
Machine Learning in Engineering Science · 19 Oct 202620:00–22:00 · 11/30
W07 · H2 · Machine Learning in Engineering Science
Retrieval is a weighted sum
Each value contributes according to its attention weight.
cᵢ = Σⱼ aᵢⱼ vⱼ
Machine Learning in Engineering Science · 19 Oct 202622:00–24:00 · 12/30
W07 · H2 · Machine Learning in Engineering Science
Read matrix axes before the formula
Rows are query destinations; columns are key sources.
(Nq × dₖ)(dₖ × Nk) → Nq × Nk
Machine Learning in Engineering Science · 19 Oct 202624:00–26:00 · 13/30
W07 · H2 · Machine Learning in Engineering Science
Work one example: choose Q, K, and V
The values are deliberately small so all intermediates can be checked.
q=[1,0] K=[[1,0],[0,1],[1,1]]
Machine Learning in Engineering Science · 19 Oct 202626:00–28:00 · 14/30
W07 · H2 · Machine Learning in Engineering Science
Work one example: scores become weights
A zero dot product still receives a positive normalized weight.
scores ≈ [0.7071, 0, 0.7071]
What are the three attention weights?
Approximately [0.4011, 0.1978, 0.4011], with sum 1.
Machine Learning in Engineering Science · 19 Oct 202628:00–30:00 · 15/30
W07 · H2 · Machine Learning in Engineering Science
Work one example: combine value coordinates
Use the same weights for every value coordinate.
context ≈ [8.0222, 5.9889]
Machine Learning in Engineering Science · 19 Oct 202630:00–32:00 · 16/30
W07 · H2 · Machine Learning in Engineering Science
Change the query; keep memory fixed
Attention can route the same memory differently for each request.
context ≈ [8.022, 5.989]
Machine Learning in Engineering Science · 19 Oct 202632:00–34:00 · 17/30
W07 · H2 · Machine Learning in Engineering Science
Self-attention uses one source sequence
Q, K, and V come from learned projections of the same representations.
Q=XWQ K=XWK V=XWV
Machine Learning in Engineering Science · 19 Oct 202634:00–36:00 · 18/30
W07 · H2 · Machine Learning in Engineering Science
Cross-attention connects two streams
Queries come from one sequence; keys and values come from another.
Q=YWQ K=XWK V=XWV
Machine Learning in Engineering Science · 19 Oct 202636:00–38:00 · 19/30
W07 · H2 · Machine Learning in Engineering Science
Causal masking hides future keys
Before softmax, set scores for key positions j>i to −∞.
allowed(i,j) ⇔ j ≤ i
Machine Learning in Engineering Science · 19 Oct 202638:00–40:00 · 20/30
W07 · H2 · Machine Learning in Engineering Science
Padding masks hide nonexistent tokens
Padding and causality enforce different constraints.
Machine Learning in Engineering Science · 19 Oct 202640:00–42:00 · 21/30
W07 · H2 · Machine Learning in Engineering Science
An all-masked row is undefined
Softmax requires at least one permitted source per query.
all scores = −∞ → no valid normalization
Machine Learning in Engineering Science · 19 Oct 202642:00–44:00 · 22/30
W07 · H2 · Machine Learning in Engineering Science
Attention exposes a short information path
A query can access a distant key within one dense attention layer.
Machine Learning in Engineering Science · 19 Oct 202644:00–46:00 · 23/30
W07 · H2 · Machine Learning in Engineering Science
Dense attention has a quadratic score grid
Doubling context length quadruples score entries at fixed head count.
per head: T × T scores
Machine Learning in Engineering Science · 19 Oct 202646:00–48:00 · 24/30
W07 · H2 · Machine Learning in Engineering Science
Open the attention notebook
Verify your hand calculation before exploring masks.
Machine Learning in Engineering Science · 19 Oct 202648:00–50:00 · 25/30
W07 · H2 · Machine Learning in Engineering Science
Test causality and normalization
Every permitted row sums to one; every future weight is zero.
row_sum(A)=1 A[i,j>i]=0
Machine Learning in Engineering Science · 19 Oct 202650:00–52:00 · 26/30
W07 · H2 · Machine Learning in Engineering Science
Compare two implementations of one operation
NumPy and PyTorch should agree for the same inputs, scale, and mask.
Machine Learning in Engineering Science · 19 Oct 202652:00–54:00 · 27/30
W07 · H2 · Machine Learning in Engineering Science
Routing weights are not causal explanations
Changing values can alter the output while attention weights remain fixed.
Machine Learning in Engineering Science · 19 Oct 202654:00–56:00 · 28/30
W07 · H2 · Machine Learning in Engineering Science
Explain the mechanism before the architecture
State the sequence: compare, normalize, retrieve.
Machine Learning in Engineering Science · 19 Oct 202656:00–58:00 · 29/30
W07 · H2 · Machine Learning in Engineering Science
Summary: compare, normalize, retrieve
Attention retrieves a query-dependent mixture of values. Masks determine which information is available.
A = softmax((QKᵀ / √dₖ) + mask) Context = AV
Machine Learning in Engineering Science · 19 Oct 202658:00–60:00 · 30/30