W09 / 2026-11-02 / 35 MIN / ENGLISH

Find failures hidden by an aggregate metric

Classroom formative practice. This does not add a weekly graded assignment. The exercise time is included in class practice, not added after the three-hour meeting.

Open exercise in Colab Download exercise notebook Worked walkthrough All exercises

Before you start

Use a CPU runtime. Save a personal copy or open the downloaded notebook in Jupyter.
Run the included complete walkthrough first. Read its setup, definitions, and checkpoints.
Predict before running each exercise cell; then change only the requested factor.
Keep your actual outputs, configuration, explanation, and limitation. Free tools are sufficient.

TASK 1 / PREDICT - RUN - EXPLAIN

Inspect subgroup errors

Predict the overall MAE when 90 cases have absolute error 1 and 10 cases have absolute error 10. Explain whose failure is hidden.

Write your prediction in the notebook response field before running.
Run the starter cell below after the worked walkthrough.
Make the requested controlled change and record the configuration.
Explain the mechanism using the actual output, not an invented result.

absolute_errors=np.r_[np.ones(90),np.full(10,10.)]
observations['task1']={'overall_MAE':float(absolute_errors.mean()),'majority_MAE':float(absolute_errors[:90].mean()),'minority_MAE':float(absolute_errors[90:].mean())}
print(observations['task1'])

Self-check: Overall MAE is 1.9 but the smaller subgroup has MAE 10. Always retain subgroup sizes.

TASK 2 / PREDICT - RUN - EXPLAIN

Stress-test a stated approximation

Use the deliberately misspecified predictor y_hat=0 for the reference y=x^2. Compare inputs in [-1,1] and [-3,3]. This is an analytic toy, not a newly trained model.

Write your prediction in the notebook response field before running.
Run the starter cell below after the worked walkthrough.
Make the requested controlled change and record the configuration.
Explain the mechanism using the actual output, not an invented result.

observations['task2']=[]
for extent in [1.,3.]:
    x_demo=np.linspace(-extent,extent,101); true_demo=x_demo**2; predicted_demo=np.zeros_like(x_demo)
    observations['task2'].append({'range':[-extent,extent],'MAE':float(np.mean(np.abs(true_demo-predicted_demo)))})
print(observations['task2'])

Self-check: Error grows as inputs move farther from zero. Report this mechanism without claiming the toy quantifies all deployment shifts.

TASK 3 / PREDICT - RUN - EXPLAIN

Choose a metric with its failure cost

Compare MAE and RMSE before and after adding one large residual. Explain which metric weights large failures more strongly.

Write your prediction in the notebook response field before running.
Run the starter cell below after the worked walkthrough.
Make the requested controlled change and record the configuration.
Explain the mechanism using the actual output, not an invented result.

small_errors=np.array([.1,-.1,.2,-.2]); outlier_errors=np.r_[small_errors,3.]
observations['task3']={name:{'MAE':float(np.mean(np.abs(a))),'RMSE':float(np.sqrt(np.mean(a**2)))} for name,a in [('small',small_errors),('with_outlier',outlier_errors)]}
print(observations['task3'])

Self-check: Both metrics change, but squared residuals make RMSE more sensitive to large failures. Keep the target units explicit.

Completion checklist

Three predictions and three explanations.
One controlled change with its actual outcome.
One limitation and a brief AI-use/checking disclosure, or "none".
A downloaded W09-evidence.json, then a two-minute peer explain-back.

The notebook marks incomplete response records as drafts. Completing the fields is not proof that an explanation is correct.

The guided walkthroughs were previously executed. The new exercise extension cells have not yet been execution-tested for this revision. Agent exercises are deterministic simulations; no paid model or consequential external action is used.

Return to this week Exercise collection