National Cheng Kung UniversityLAiMM Laboratory

W05 / 2026-10-05 / 40 MIN / ENGLISH

Confusion matrices, thresholds, and prevalence

Classroom formative practice. This does not add a weekly graded assignment. The exercise time is included in class practice, not added after the three-hour meeting.

Open exercise in ColabDownload exercise notebookWorked walkthroughAll exercises

Before you start

  1. Use a CPU runtime. Save a personal copy or open the downloaded notebook in Jupyter.
  2. Run the included complete walkthrough first. Read its setup, definitions, and checkpoints.
  3. Predict before running each exercise cell; then change only the requested factor.
  4. Keep your actual outputs, configuration, explanation, and limitation. Free tools are sufficient.

TASK 1 / PREDICT - RUN - EXPLAIN

Explain every denominator

Compute precision, recall/sensitivity, specificity, and accuracy from the counts below. Name the population in each denominator before running.

  1. Write your prediction in the notebook response field before running.
  2. Run the starter cell below after the worked walkthrough.
  3. Make the requested controlled change and record the configuration.
  4. Explain the mechanism using the actual output, not an invented result.
tp,fn,fp,tn=18,2,8,72
observations['task1']={'precision':tp/(tp+fp),'recall_sensitivity':tp/(tp+fn),'specificity':tn/(tn+fp),'accuracy':(tp+tn)/(tp+tn+fp+fn)}
print(observations['task1'])

Self-check: Recall is 0.9, precision is 18/26, specificity is 0.9, and accuracy is 0.9. Precision and recall answer different questions.

TASK 2 / PREDICT - RUN - EXPLAIN

Choose using validation, not test

The following labels and probabilities are a validation set. Compare two thresholds, describe the cost of a false negative, and declare a choice before consulting any test set.

  1. Write your prediction in the notebook response field before running.
  2. Run the starter cell below after the worked walkthrough.
  3. Make the requested controlled change and record the configuration.
  4. Explain the mechanism using the actual output, not an invented result.
from sklearn.metrics import confusion_matrix
validation_y=np.array([0,0,0,1,1,1])
validation_probability=np.array([.1,.35,.6,.4,.7,.9])
observations['task2']={}
for threshold_demo in [.3,.7]:
    cm=confusion_matrix(validation_y,validation_probability>=threshold_demo,labels=[0,1])
    observations['task2'][str(threshold_demo)]=cm.tolist()
print(observations['task2'])

Self-check: At 0.3 there are no false negatives and two false positives. At 0.7 there is one false negative and no false positives. Explain the tradeoff.

TASK 3 / PREDICT - RUN - EXPLAIN

Change prevalence without changing rates

Assume sensitivity 0.9 and false-positive rate 0.1. Compare expected counts in 1000 cases at prevalence 0.1 and 0.01. These are mathematical expectations, not measured model results.

  1. Write your prediction in the notebook response field before running.
  2. Run the starter cell below after the worked walkthrough.
  3. Make the requested controlled change and record the configuration.
  4. Explain the mechanism using the actual output, not an invented result.
observations['task3']=[]
for prevalence in [.1,.01]:
    expected_tp=1000*prevalence*.9
    expected_fp=1000*(1-prevalence)*.1
    observations['task3'].append({'prevalence':prevalence,'expected_TP':expected_tp,'expected_FP':expected_fp,'expected_precision':expected_tp/(expected_tp+expected_fp)})
print(observations['task3'])

Self-check: Precision falls from 0.5 to about 0.0833 even though sensitivity and false-positive rate are held fixed.

Completion checklist

The notebook marks incomplete response records as drafts. Completing the fields is not proof that an explanation is correct.

The guided walkthroughs were previously executed. The new exercise extension cells have not yet been execution-tested for this revision. Agent exercises are deterministic simulations; no paid model or consequential external action is used.

Return to this weekExercise collection