{
  "nbformat": 4,
  "nbformat_minor": 5,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.11"
    },
    "colab": {
      "name": "week-15-exercise.ipynb"
    },
    "course_exercise": {
      "id": "W15",
      "week": "15",
      "hour": null,
      "date": "2026-12-14",
      "title": "A traceable bounded analysis request",
      "minutes": 35,
      "optional": false,
      "worksheet": "notes/exercises/week-15-exercise.html",
      "notebook": "notebooks/exercises/week-15-exercise.ipynb",
      "colab": "https://colab.research.google.com/github/CHjonnyYu/MLES-Fall-2026/blob/main/notebooks/exercises/week-15-exercise.ipynb",
      "guided_notebook": "notebooks/week-15-practice.ipynb",
      "execution_status": "New exercise extension not executed in this authoring run"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "# W15 Exercise: A traceable bounded analysis request\n\nMachine Learning in Engineering Science | NCKU / LAiMM | Fall 2026\n\n**Date:** 2026-12-14. **Exercise time:** 35 minutes within the class/practice allocation, not extra time after class.\n\n**Classroom formative practice. No new weekly graded submission is created.**\n\n## How to work independently\n1. In Colab, save a copy to your Drive and use a CPU runtime. In Jupyter, open this notebook and select a Python kernel.\n2. Run the complete worked walkthrough below from top to bottom. Read every Markdown checkpoint; the walkthrough is included so no previous notebook state is needed.\n3. At the Independent Exercise section, write each prediction in the response cell before running its experiment.\n4. Run the starter experiment, make the requested controlled change, and record the actual result. Edit only the declared factor; preserve split rules and preprocessing.\n5. Explain one mechanism and one limitation. Export the evidence JSON and discuss it with a partner.\n\nThe core exercises need no paid API, private dataset, or GPU. The agent units use clearly labeled offline simulators, not real LLM calls.\n\n**Reproducibility boundary:** the original guided notebooks were previously executed; the new extension cells have not yet been execution-tested in this revision. Report unexpected behavior rather than claiming a check passed.\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "# Week 15: Agent Studio I: a bounded analysis workflow\n\nMachine Learning in Engineering Science | NCKU + LAiMM | Fall 2026\n\n**Purpose:** a fully worked, self-paced in-class practice with an optional extension. Weeks 11 and 18 are optional examination-readiness materials. These notebooks do not create weekly graded assignments.\n\n**Before running:** choose a Python 3 kernel in Jupyter or a CPU runtime in Colab. Run every cell from top to bottom with Shift+Enter. Setup installs missing free packages. No API key or external dataset download is required.\n\n**Learning target:** complete the calculation, inspect intermediate values, and explain why the protocol supports a limited claim.\n\n**Time:** about 25-45 minutes including reading, prediction, execution, and explain-back.\n",
      "id": "cell-000"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## 1. Set up and record the environment\n\nKeep the seed fixed for the first run. Record actual outputs and package versions rather than copying a claimed performance number.\n",
      "id": "cell-001"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "metadata": {},
      "source": "import sys, subprocess, importlib.util, platform, json, random, hashlib\nfrom pathlib import Path\nrequired = {\"numpy\": \"numpy\", \"matplotlib\": \"matplotlib\", \"sklearn\": \"scikit-learn\"}\nmissing = [package for module, package in required.items() if importlib.util.find_spec(module) is None]\nif missing:\n    subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", *missing])\nimport numpy as np\nimport matplotlib.pyplot as plt\nSEED = 2026\nrandom.seed(SEED)\nnp.random.seed(SEED)\nrng = np.random.default_rng(SEED)\nARTIFACTS = Path(\"artifacts\")\nARTIFACTS.mkdir(exist_ok=True)\nprint(\"Python:\", platform.python_version(), \"NumPy:\", np.__version__, \"Seed:\", SEED)\n",
      "id": "cell-002"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "**Checkpoint.** The environment cell must finish before any analysis cell. Restart and run all if a variable is missing.\n",
      "id": "cell-003"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## 2. Define a structured measurement request\n\nThis deterministic workflow is a baseline for an engineering assistant. It accepts numeric readings and returns summary evidence without inventing a physical diagnosis.\n",
      "id": "cell-004"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "metadata": {},
      "source": "request={\"task\":\"summarize\",\"units\":\"synthetic units\",\"readings\":[3.1,3.2,3.0,3.6,3.4]}\ndef summarize(request):\n    values=request.get(\"readings\")\n    if request.get(\"task\")!=\"summarize\" or not isinstance(values,list) or len(values)<2:\n        raise ValueError(\"Need a summarize request with at least two readings\")\n    if not all(type(x) in (int,float) and np.isfinite(x) for x in values): raise ValueError(\"Need finite numbers\")\n    return {\"n\":len(values),\"mean\":float(np.mean(values)),\"sample_std\":float(np.std(values,ddof=1)),\"units\":request[\"units\"]}\nresult=summarize(request); print(result)\n",
      "id": "cell-005"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "**Checkpoint.** The sample standard deviation uses ddof=1. Units remain part of the result record.\n",
      "id": "cell-006"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## 3. Trace every operation\n\nA reproducible workflow can be independently rerun. Preserve the exact input and a deterministic digest.\n",
      "id": "cell-007"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "metadata": {},
      "source": "payload=json.dumps(request,sort_keys=True).encode()\ntrace={\"request_sha256\":hashlib.sha256(payload).hexdigest(),\"request\":request,\"result\":result,\n       \"limitation\":\"descriptive statistics; no calibrated diagnostic threshold\"}\nassert summarize(request)==result\nprint(trace[\"request_sha256\"])\n",
      "id": "cell-008"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "**Checkpoint.** The hash identifies the request; it does not certify measurement authenticity.\n",
      "id": "cell-009"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## 4. Test a malformed request\n\nA failed input check is a defined behavior. Record the error rather than treating it as a model hallucination.\n",
      "id": "cell-010"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "metadata": {},
      "source": "try:\n    summarize({\"task\":\"summarize\",\"units\":\"u\",\"readings\":[\"unknown\",2]})\nexcept ValueError as error:\n    failure=str(error)\nelse:\n    raise AssertionError(\"Invalid request should fail\")\nsummary={\"trace\":trace,\"invalid_input_error\":failure}\nprint(summary[\"invalid_input_error\"])\n",
      "id": "cell-011"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "**Checkpoint.** A later LLM interface should produce this same schema and must still pass the validator.\n",
      "id": "cell-012"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## 6. Preserve your evidence\n\nExport a JSON record with a seed, package version, and measured result. Download the artifacts directory before leaving Colab; runtime storage is temporary.\n",
      "id": "cell-013"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "metadata": {},
      "source": "record={\"week\":15,\"seed\":SEED,\"numpy\":np.__version__,\"python\":platform.python_version(),\"result\":summary}\n(ARTIFACTS/\"week-15-evidence.json\").write_text(json.dumps(record,indent=2))\nprint(json.dumps(record,indent=2))\n",
      "id": "cell-014"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "**Checkpoint.** The evidence file must match the result you actually obtained. A saved screenshot alone does not permit rerunning the calculation.\n",
      "id": "cell-015"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## 7. Try one change and explain it\n1. Predict the effect of one change before editing code. Change one parameter, input, or boundary condition only.\n2. Rerun all cells in a fresh kernel. Keep the original evidence record and compare the observed difference.\n3. Explain the result using the mechanism, not just the final number.\n4. Name one assumption that limits the claim, and a new test that could challenge it.\n\n**Troubleshooting:** if installation fails, restore network access; if a variable is undefined, run earlier cells; if your output differs, compare package versions, seed, and data construction. Do not repeatedly edit code until a preferred score appears.\n\n**Completion criterion:** one executed notebook, one evidence JSON, and a brief claim/evidence/limitation statement.\n\n### Reference documentation\n- [NumPy user guide](https://numpy.org/doc/stable/user/)\n- [scikit-learn user guide](https://scikit-learn.org/stable/user_guide.html)\n- [Course repository](https://github.com/CHjonnyYu/MLES-Fall-2026)\n",
      "id": "cell-016"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## Independent Exercise: W15\n\nComplete the three tasks below after the walkthrough. The starter cells produce observations, not your written interpretation. Fill the response fields in the next cell. Save separate records for the original run and any controlled change."
    },
    {
      "cell_type": "code",
      "metadata": {},
      "execution_count": null,
      "outputs": [],
      "source": "exercise_id=\"W15\"\nobservations={}\nresponses={\n    'task1_prediction':'', 'task1_explanation':'',\n    'task2_prediction':'', 'task2_explanation':'',\n    'task3_prediction':'', 'task3_explanation':'',\n    'controlled_change':'', 'limitation':'', 'ai_use_and_checks':''\n}\n# Fill these strings before and after each experiment. Use 'none' if no AI assistance was used.\nprint('Exercise record initialized:', exercise_id)"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "### Task 1: Explain the standard-deviation convention\n\nCalculate the mean and population/sample standard deviations for the readings. Explain ddof=1 without claiming it makes these measurements authentic.\n\n**Steps**\n1. Write your prediction in `responses['task1_prediction']` before running.\n2. Run the starter cell; read every printed quantity and retain the observation.\n3. Make the controlled comparison requested above. Record its configuration in `responses['controlled_change']`.\n4. Write a mechanism-based explanation in `responses['task1_explanation']`; do not merely repeat a score.\n\n**Checkpoint:** The mean is 3.26. Sample standard deviation uses denominator n-1; units must accompany the result."
    },
    {
      "cell_type": "code",
      "metadata": {},
      "execution_count": null,
      "outputs": [],
      "source": "readings_ex=np.array([3.1,3.2,3.0,3.6,3.4])\nobservations['task1']={'n':len(readings_ex),'mean':float(readings_ex.mean()),'population_sd':float(readings_ex.std(ddof=0)),'sample_sd':float(readings_ex.std(ddof=1)),'units':'synthetic units'}\nprint(observations['task1'])"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "### Task 2: Hash a request including its units\n\nKeep numerical readings fixed but change the declared unit string. Explain why the request identity changes.\n\n**Steps**\n1. Write your prediction in `responses['task2_prediction']` before running.\n2. Run the starter cell; read every printed quantity and retain the observation.\n3. Make the controlled comparison requested above. Record its configuration in `responses['controlled_change']`.\n4. Write a mechanism-based explanation in `responses['task2_explanation']`; do not merely repeat a score.\n\n**Checkpoint:** The hashes differ. Changing a unit label without converting measurements is not a scientifically valid unit conversion."
    },
    {
      "cell_type": "code",
      "metadata": {},
      "execution_count": null,
      "outputs": [],
      "source": "request_a={'task':'summarize','units':'synthetic units','readings':readings_ex.tolist()}\nrequest_b={**request_a,'units':'different declared units'}\ndigest_a=hashlib.sha256(json.dumps(request_a,sort_keys=True).encode()).hexdigest()\ndigest_b=hashlib.sha256(json.dumps(request_b,sort_keys=True).encode()).hexdigest()\nobservations['task2']={'request_a_sha256':digest_a,'request_b_sha256':digest_b,'same_request_identity':digest_a==digest_b}\nprint(observations['task2'])"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "### Task 3: Refuse malformed numeric data\n\nCall the worked validator with a nonnumeric reading. Explain why an LLM-generated request must pass the same check.\n\n**Steps**\n1. Write your prediction in `responses['task3_prediction']` before running.\n2. Run the starter cell; read every printed quantity and retain the observation.\n3. Make the controlled comparison requested above. Record its configuration in `responses['controlled_change']`.\n4. Write a mechanism-based explanation in `responses['task3_explanation']`; do not merely repeat a score.\n\n**Checkpoint:** The malformed request should be rejected. Record any unexpected behavior rather than presenting it as a passed check."
    },
    {
      "cell_type": "code",
      "metadata": {},
      "execution_count": null,
      "outputs": [],
      "source": "try:\n    malformed_result=summarize({'task':'summarize','units':'u','readings':['unknown',2]})\n    observations['task3']={'rejected':False,'observed':malformed_result}\nexcept ValueError as error:\n    observations['task3']={'rejected':True,'reason':str(error)}\nprint(observations['task3'])"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "### Finish: preserve and explain\n\nFill all response strings, then run the export cell. A draft file is intentionally labeled incomplete; a complete response record is not automatically a scientifically correct result. Download the JSON before the Colab session ends. Share it with a partner for explain-back, not as an additional weekly submission.\n\n**Exit questions:** What changed? Which observation supports your explanation? What does this experiment not establish?"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "execution_count": null,
      "outputs": [],
      "source": "missing_responses=[key for key,value in responses.items() if not str(value).strip()]\nexercise_record={'exercise':exercise_id,'status':'draft' if missing_responses else 'ready for peer discussion','responses':responses,'observations':observations,'missing_responses':missing_responses,'environment':{'python':platform.python_version(),'numpy':np.__version__},'data_scope':'course-provided or synthetic teaching data'}\nexercise_folder=Path('artifacts')/'exercises'; exercise_folder.mkdir(parents=True,exist_ok=True)\nexercise_path=exercise_folder/(exercise_id+'-evidence.json')\nexercise_path.write_text(json.dumps(exercise_record,indent=2,default=lambda value:value.tolist() if hasattr(value,'tolist') else str(value)))\nprint('Saved:',exercise_path)\nprint('Status:',exercise_record['status'])\nprint('Fields still to complete:',missing_responses)"
    }
  ]
}