Define a valid sequence task, then explain and test how LSTM stores information.

Goal: trace one cell update and judge a forecast.

A sequence contains both values and an ordering rule.

[1, 2, 4, 8] ≠ [8, 1, 4, 2]

What remains identical after shuffling?

The multiset and mean; adjacency, trend, and time-dependent structure can change.

The prediction must use only information available at its origin.

history: x[t−L:t] target: x[t]

Sequence models receive a batch of histories, with features at every time step.

X ∈ ℝᴮˣᵀˣᶠ

Many-to-one predicts one value; many-to-many predicts a value at multiple positions.

Train on earlier targets, select on later validation, then evaluate on test.

train < validation < test

Global means and variances can reveal future distribution information.

z = (x − μtrain) / σtrain

Predict the next observation using the last known observation.

ŷ[t] = x[t−1]

An RNN applies the same state-update parameters at every time step.

hₜ = tanh(Wₓxₜ + Wₕhₜ₋₁ + b)

Backpropagation through time multiplies Jacobians along earlier states.

∂hₜ/∂hₜ₋ₖ = Jₜ Jₜ₋₁ … Jₜ₋ₖ₊₁

The cell state carries memory; the hidden state exposes a filtered view.

cell: cₜ hidden output: hₜ

Sigmoid outputs values between zero and one for each hidden dimension.

σ(z) = 1 / (1 + exp(−z))

Multiply the previous cell state by a learned forget coefficient.

fₜ = σ(Wf [hₜ₋₁; xₜ] + bf)

A candidate proposes content; the input gate scales how much is written.

iₜ = σ(Wi [hₜ₋₁; xₜ] + bi) gₜ = tanh(Wg [hₜ₋₁; xₜ] + bg)

Retain selected old memory and add selected new content.

cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ gₜ

The hidden state is a bounded, gated view of the updated cell.

oₜ = σ(Wo [hₜ₋₁; xₜ] + bo) hₜ = oₜ ⊙ tanh(cₜ)

Chosen gates make every intermediate step visible.

0.9 × 0.8 + 0.2 × (−0.5) = 0.62

What is the hidden output?

h = 0.7 × tanh(0.62) ≈ 0.386.

A high forget coefficient does not prevent new content from changing memory.

cnew = f × cold + i × g

Run setup and inspect the synthetic sensor before fitting anything.

A correct shape is insufficient if the window contains its target.

X[i] = signal[t−24:t] y[i] = signal[t]

Ridge uses fixed window positions; LSTM updates a shared recurrent state.

batch_first changes input/output layout, not hidden-state layout.

input: (B,T,1) states: (B,T,H) hₙ: (layers,B,H)

Gradient updates use training; checkpoint selection uses validation.

Inspect actual training and validation trajectories in your notebook.

MAE averages error magnitude; RMSE emphasizes large residuals.

MAE = mean |ŷ−y| RMSE = √mean(ŷ−y)²

PyTorch concatenates its learned gate parameters in i, f, g, o order.

linear projections → split → activations → cell update

Change window length while preserving the forecast question and split boundaries.

Retention depends on learned gates, sequence structure, and optimization.

direct memory factor ≈ ∏ fⱼ