National Cheng Kung UniversityLAiMM Laboratory

W07-H3 / 2026-10-19 / 25 MIN / ENGLISH

Trace and test a tiny causal Transformer

Classroom formative practice. This does not add a weekly graded assignment. The exercise time is included in class practice, not added after the three-hour meeting.

Open exercise in ColabDownload exercise notebookWorked walkthroughAll exercises

Before you start

  1. Use a CPU runtime. Save a personal copy or open the downloaded notebook in Jupyter.
  2. Run the included complete walkthrough first. Read its setup, definitions, and checkpoints.
  3. Predict before running each exercise cell; then change only the requested factor.
  4. Keep your actual outputs, configuration, explanation, and limitation. Free tools are sufficient.

TASK 1 / PREDICT - RUN - EXPLAIN

Trace model shapes

After the guided training run, inspect a two-example batch. Explain the final axis and why logits are not probabilities.

  1. Write your prediction in the notebook response field before running.
  2. Run the starter cell below after the worked walkthrough.
  3. Make the requested controlled change and record the configuration.
  4. Explain the mechanism using the actual output, not an invented result.
model.eval()
with torch.no_grad():
    ex_logits=model(test_x[:2])
observations['task1']={'input_shape':list(test_x[:2].shape),'logit_shape':list(ex_logits.shape),'vocabulary_size':VOCAB}
print(observations['task1'])

Self-check: Input shape is (2,16) and logits have shape (2,16,8). Each position predicts a distribution over the next token.

TASK 2 / PREDICT - RUN - EXPLAIN

Change future positions only

Keep positions 0 through 7 fixed. Change positions 8 through 15 using modulo VOCAB, then compare earlier logits. Explain why model.eval() matters.

  1. Write your prediction in the notebook response field before running.
  2. Run the starter cell below after the worked walkthrough.
  3. Make the requested controlled change and record the configuration.
  4. Explain the mechanism using the actual output, not an invented result.
original_ex=test_x[:1].clone(); changed_ex=original_ex.clone()
changed_ex[:,8:]=(changed_ex[:,8:]+3)%VOCAB
with torch.no_grad():
    original_logits=model(original_ex); changed_logits=model(changed_ex)
delta_ex=float((original_logits[:,:8]-changed_logits[:,:8]).abs().max())
observations['task2']={'unchanged_positions':list(range(8)),'changed_positions':list(range(8,16)),'max_earlier_logit_difference':delta_ex,'passes_1e_minus_6':delta_ex<1e-6}
print(observations['task2'])

Self-check: The earlier logits should agree within the declared tolerance. Token values remain in 0 through 7. This is a behavioral causality check, not a model-accuracy result.

TASK 3 / PREDICT - RUN - EXPLAIN

Explain sequential generation

Generate five new symbols using greedy selection. Explain why training can score many positions at once while this generation loop is sequential.

  1. Write your prediction in the notebook response field before running.
  2. Run the starter cell below after the worked walkthrough.
  3. Make the requested controlled change and record the configuration.
  4. Explain the mechanism using the actual output, not an invented result.
prefix_ex=test_x[:1,:4].clone()
for _ in range(5):
    with torch.no_grad():
        next_ex=model(prefix_ex[:,-CONTEXT:])[:,-1].argmax(dim=-1,keepdim=True)
    prefix_ex=torch.cat([prefix_ex,next_ex],dim=1)
observations['task3']={'generated_ids':prefix_ex[0].tolist(),'new_symbols':5,'decoding':'greedy'}
print(observations['task3'])

Self-check: Each generated symbol becomes part of the next input. This tiny synthetic model does not demonstrate natural-language understanding or autonomous tool use.

Completion checklist

The notebook marks incomplete response records as drafts. Completing the fields is not proof that an explanation is correct.

The guided walkthroughs were previously executed. The new exercise extension cells have not yet been execution-tested for this revision. Agent exercises are deterministic simulations; no paid model or consequential external action is used.

Return to this weekExercise collection