{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "# Homework: Build and Explain a Stock Forecasting System\n**Machine Learning in Engineering Science | NCKU + LAiMM | Fall 2026**\n\n[Read the full assignment](https://mles-course-vgqg4abvka-de.a.run.app/notes/homework/stock-market-prediction.html)\n\n## Start here\n1. Save a copy to your Drive (Colab) or download this notebook (Jupyter).\n2. Select a CPU runtime. No paid API, trading account, or GPU is required.\n3. Read the Markdown before running each cell, in order.\n4. Choose your stock/ETF and document its data source. This is **not** a trading recommendation.\n5. Study the worked scaffolding, complete the student explanations, and justify your decisions. Running unchanged code is not a complete submission.\n6. Keep the final test gate closed until you have selected your model using validation data.\n\n**Publication status:** this is an unexecuted starter, not a validated result. There are no precomputed performance claims. Download availability and package behavior can vary; preserve your actual environment and dataset. If online retrieval fails, use the documented CSV-upload route.\n\n**Deliverables:** executed notebook, reproducibility files, original-unit predictions and metrics, interpretation, and a **five-minute English video**. Due date: instructor announcement.\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## 1. Prediction contract and student decisions\nAt the close of trading observation t, predict the close at the next trading observation t+1 using only information available by t. With L observations, the input is [P(t-L+1), ..., P(t)] and the target is P(t+1).\n\n**Task A:** Before running, write what information is available, why a random split is inappropriate, and what a persistence model predicts. Compare lookbacks 10 and 20 only on validation data. Test results are not tuning feedback.\n\nThe sample ticker below is a demonstration, not a recommended investment. Change it if you prefer. The fixed end date excludes incomplete current-day prices. For uploaded data, supply provenance and the adjustment convention yourself.\n"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": "import os, sys, subprocess, importlib.util\nos.environ['CUDA_VISIBLE_DEVICES'] = '-1'  # CPU-first teaching example; set before TensorFlow import.\nos.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'\npackages = {'numpy':'numpy', 'pandas':'pandas', 'matplotlib':'matplotlib',\n            'sklearn':'scikit-learn', 'tensorflow':'tensorflow', 'yfinance':'yfinance', 'joblib':'joblib'}\nmissing = [package for module, package in packages.items() if importlib.util.find_spec(module) is None]\nif missing:\n    subprocess.check_call([sys.executable, '-m', 'pip', 'install', *missing])\n\nimport json, hashlib, random, platform, shutil\nfrom pathlib import Path\nfrom datetime import datetime, timezone\nfrom importlib.metadata import version\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport tensorflow as tf\nimport yfinance as yf\nimport joblib\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.linear_model import Ridge\n\nSEED = 2026\nrandom.seed(SEED)\nnp.random.seed(SEED)\ntf.keras.utils.set_random_seed(SEED)\nWORK = Path('stock_hw_work')\nDATA = WORK / 'data'\nFIGURES = WORK / 'figures'\nfor folder in (WORK, DATA, FIGURES):\n    folder.mkdir(parents=True, exist_ok=True)\nenvironment = {'python': platform.python_version(), **{p: version(p) for p in packages.values()}}\n(WORK / 'environment.json').write_text(json.dumps(environment, indent=2))\n(WORK / 'requirements.txt').write_text('\\n'.join(f'{p}=={version(p)}' for p in packages.values()) + '\\n')\nprint(environment)\nprint('A seed documents the experiment; exact results can still vary by platform/runtime.')\n"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": "# STUDENT: choose one instrument and explain your choice in the next Markdown cell.\nTICKER = 'SPY'\nSTART = '2021-01-01'\nEND = '2026-10-17'  # Exclusive for yfinance; no unfinished current trading session.\nMODE = 'download'  # 'download' or 'upload'\nCURRENCY = 'USD'  # Change for other markets; this is a student declaration.\nUPLOAD_PATH = ''  # Local Jupyter: path to your CSV. Colab: leave blank to show upload dialog.\nUPLOAD_SOURCE_URL = ''  # Required in upload mode.\nUPLOAD_PRICE_CONVENTION = ''  # Example: vendor-adjusted close; explain exactly what it means.\nLOOKBACKS = (10, 20)\nHIDDEN_UNITS = 16\nEPOCHS = 20\nBATCH_SIZE = 32\nLEARNING_RATE = 0.001\nRIDGE_ALPHAS = (0.1, 1.0, 10.0)\nTRAIN_RATIO, VALID_RATIO = 0.70, 0.15\nANSWERS = {\n    'prediction_contract_and_ticker': '',\n    'leakage_prevention': '',\n    'window_alignment': '',\n    'validation_choice': '',\n    'why_lstm_won_or_lost': '',\n    'largest_error_analysis': '',\n    'interpretability_and_limitations': '',\n    'ai_assistance_and_personal_checks': '',\n    'video_url_or_filename': ''\n}\nprint('Fill ANSWERS with your own explanations before exporting your final submission.')\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "### Your prediction contract\n**Replace this paragraph:** identify the instrument, market/currency, forecast horizon, available features, and why you chose this dataset. State your expectations **before** seeing results. You may also put this explanation into ANSWERS above.\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## 2. Acquire a frozen dataset, with an upload fallback\n**Download route:** fetch daily adjusted closing prices through yfinance; its Yahoo-based service is not an official exchange feed and may be unavailable or rate-limited. Do not repeatedly retry a failing service.\n\n**Upload route:** set MODE to 'upload', fill the source URL/price convention/currency, then upload a CSV with columns **date, close** (case-insensitive). Obtain at least **750 trading observations** from a source you may use. Never substitute synthetic prices for the final experiment.\n\nThe original provider file and a canonical two-column snapshot are saved separately. Later runs reuse the snapshot rather than silently changing the data. To study a different acquisition, use a separate working folder and preserve both experiments. Do not assume that retrospective adjustments are point-in-time historical information. Respect redistribution restrictions when sharing files.\n"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": "snapshot = DATA / 'prices.csv'\nprovenance_path = DATA / 'provenance.json'\nrequest_identity = {'ticker': TICKER, 'start': START, 'end_exclusive': END, 'mode': MODE,\n                    'currency': CURRENCY, 'upload_source_url': UPLOAD_SOURCE_URL,\n                    'upload_price_convention': UPLOAD_PRICE_CONVENTION}\nif snapshot.exists():\n    if not provenance_path.exists():\n        raise ValueError('Snapshot lacks provenance. Restore its metadata rather than guessing its source.')\n    provenance = json.loads(provenance_path.read_text())\n    if provenance['request'] != request_identity:\n        raise ValueError('Configuration differs from the saved acquisition. Use a separate WORK folder.')\n    df = pd.read_csv(snapshot)\n    print('Reusing saved snapshot:', snapshot)\nelse:\n    if MODE == 'download':\n        raw = yf.download(TICKER, start=START, end=END, interval='1d',\n                          auto_adjust=True, actions=False, progress=False,\n                          threads=False, multi_level_index=False, timeout=20)\n        if raw is None or raw.empty:\n            raise RuntimeError('No prices returned. Use MODE=upload with a permitted CSV source.')\n        original_path = DATA / 'provider_download.csv'\n        raw.to_csv(original_path)\n        df = pd.DataFrame({'date': raw.index, 'close': raw['Close'].to_numpy()})\n        source_url = 'https://finance.yahoo.com/quote/' + TICKER + '/history/'\n        price_convention = 'yfinance auto_adjust=True: vendor-adjusted closing price'\n    elif MODE == 'upload':\n        if not UPLOAD_SOURCE_URL or not UPLOAD_PRICE_CONVENTION:\n            raise ValueError('Fill UPLOAD_SOURCE_URL and UPLOAD_PRICE_CONVENTION first.')\n        original_path = DATA / 'uploaded_original.csv'\n        if UPLOAD_PATH:\n            original_path.write_bytes(Path(UPLOAD_PATH).read_bytes())\n        else:\n            try:\n                from google.colab import files\n            except ImportError as exc:\n                raise RuntimeError('Local Jupyter: set UPLOAD_PATH to your CSV file.') from exc\n            uploaded = files.upload()\n            if len(uploaded) != 1:\n                raise ValueError('Upload exactly one CSV.')\n            original_path.write_bytes(next(iter(uploaded.values())))\n        df = pd.read_csv(original_path)\n        df.columns = [str(c).strip().lower() for c in df.columns]\n        if not {'date', 'close'}.issubset(df.columns):\n            raise ValueError('Your CSV needs date and close columns. Document any column conversion.')\n        df = df[['date', 'close']].copy()\n        source_url, price_convention = UPLOAD_SOURCE_URL, UPLOAD_PRICE_CONVENTION\n    else:\n        raise ValueError('MODE must be download or upload.')\n    df.to_csv(snapshot, index=False)\n    provenance = {'request': request_identity, 'source_url': source_url,\n                  'retrieved_utc': datetime.now(timezone.utc).isoformat(),\n                  'price_convention': price_convention,\n                  'original_file': original_path.name,\n                  'original_sha256': hashlib.sha256(original_path.read_bytes()).hexdigest(),\n                  'snapshot_sha256': hashlib.sha256(snapshot.read_bytes()).hexdigest()}\n    provenance_path.write_text(json.dumps(provenance, indent=2))\n\nactual_hash = hashlib.sha256(snapshot.read_bytes()).hexdigest()\nif actual_hash != provenance['snapshot_sha256']:\n    raise ValueError('The saved snapshot changed. Document a new version; do not silently reuse old provenance.')\nprint(json.dumps(provenance, indent=2))\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## 3. Inspect data before modeling\nThis scaffold sorts dates but does **not** silently drop duplicates, fill missing prices, or invent observations. If a check stops you, inspect the original data, document your decision, and create a traceable corrected input. A source download and a cleaned table are different artifacts.\n\n**Checkpoint:** explain missing trading dates, price units, adjustment conventions, and any corporate-action limitations. Do not interpolate using a future observation.\n"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": "df['date'] = pd.to_datetime(df['date'], errors='raise').dt.tz_localize(None)\ndf['close'] = pd.to_numeric(df['close'], errors='raise')\nif df[['date', 'close']].isna().any().any():\n    raise ValueError('Missing date/close: inspect and document a cleaning decision.')\nif df['date'].duplicated().any():\n    raise ValueError('Duplicate trading dates: inspect and resolve explicitly.')\nif not np.isfinite(df['close']).all() or (df['close'] <= 0).any():\n    raise ValueError('Closing prices must be finite and positive.')\nwas_sorted = df['date'].is_monotonic_increasing\ndf = df.sort_values('date').reset_index(drop=True)\nif len(df) < 750:\n    raise ValueError(f'Found {len(df)} observations; at least 750 are required.')\ndf.to_csv(DATA / 'cleaned_prices.csv', index=False)\nprint('Already sorted:', was_sorted, '| rows:', len(df), '| currency:', CURRENCY)\nprint(df.head().to_string(index=False))\nprint(df.tail().to_string(index=False))\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## 4. Split observations chronologically and fit preprocessing on training only\nWe assign examples using their **target row/date**. Training is the first 70% of observations, validation the next 15%, and test the last 15%. Rounding is shown explicitly.\n\nOnly training observations fit the scaler. Transforming a held-out value with a training-fitted scaler is permitted; letting a held-out value determine that scaler is not. The test data will not be used for choosing model settings.\n"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": "n = len(df)\ntrain_end = int(n * TRAIN_RATIO)\nvalid_end = int(n * (TRAIN_RATIO + VALID_RATIO))\nprices = df['close'].to_numpy(dtype=np.float64)\nscaler = StandardScaler().fit(prices[:train_end].reshape(-1, 1))\nscaled = scaler.transform(prices.reshape(-1, 1)).astype(np.float32)\nsplit_record = []\nfor label, start, stop in [('train', 0, train_end), ('validation', train_end, valid_end), ('test', valid_end, n)]:\n    split_record.append({'split':label, 'rows':stop-start,\n                         'first_date':str(df.loc[start,'date'].date()),\n                         'last_date':str(df.loc[stop-1,'date'].date())})\nprint(pd.DataFrame(split_record).to_string(index=False))\n(WORK / 'split_dates.json').write_text(json.dumps(split_record, indent=2))\nfig, ax = plt.subplots(figsize=(12,4))\nax.plot(df['date'], prices, color='#142b3b', linewidth=1)\nfor label, start, stop, color in [('Train',0,train_end,'#b6d8cd'),\n                                 ('Validation',train_end,valid_end,'#edd7a5'), ('Test',valid_end,n,'#d9c4d4')]:\n    ax.axvspan(df.loc[start,'date'], df.loc[stop-1,'date'], alpha=.45, color=color, label=label)\nax.set(title=f'{TICKER}: chronological observation splits', ylabel=f'Closing price ({CURRENCY})')\nax.legend(); fig.tight_layout(); fig.savefig(FIGURES / '01_splits.png'); plt.show()\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## 5. Construct windows and prove their alignment\nFor target row j, input rows are j-L through j-1. Row j must **never** appear in its own input. Both lookbacks use the same target-row set, starting at row 20, so the controlled comparison changes history length rather than evaluation dates.\n\nA validation window may include earlier training observations. Later test windows may include already observed earlier test prices: that is rolling **one-step** forecasting, not a month-ahead recursive forecast. Model weights remain fixed during test evaluation.\n\n**Task B:** Use the displayed example to explain the array shape (examples, time steps, features) and the difference between input date and target date.\n"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": "targets = np.arange(max(LOOKBACKS), n)\ntrain_mask = targets < train_end\nvalid_mask = (targets >= train_end) & (targets < valid_end)\ntest_mask = targets >= valid_end\ny_scaled = scaled[targets, 0]\ny_original = prices[targets]\nwindows = {L: np.stack([scaled[j-L:j] for j in targets]) for L in LOOKBACKS}\nfor L, X in windows.items():\n    print('L=', L, 'shape=', X.shape, 'train/validation/test=',\n          (int(train_mask.sum()), int(valid_mask.sum()), int(test_mask.sum())))\nexample_j = int(targets[valid_mask][0])\nexample_dates = df.loc[example_j-20:example_j-1, ['date','close']]\nprint('Example: first validation target')\nprint(example_dates.to_string(index=False))\nprint('Target:', df.loc[example_j, ['date','close']].to_dict())\nprint('Input ends before target:', df.loc[example_j-1,'date'] < df.loc[example_j,'date'])\n\ndef original_units(values):\n    return scaler.inverse_transform(np.asarray(values).reshape(-1,1)).ravel()\n\ndef scores(truth, prediction):\n    residual = np.asarray(prediction).ravel() - np.asarray(truth).ravel()\n    return {'MAE':float(np.mean(np.abs(residual))), 'RMSE':float(np.sqrt(np.mean(residual**2)))}\n\nbaseline_valid = prices[targets[valid_mask]-1]\nvalidation_baseline = scores(y_original[valid_mask], baseline_valid)\nprint('Persistence validation:', validation_baseline)\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## 6. Train and select a Ridge baseline using validation data\nRidge sees the same 20 past closing prices, flattened into 20 features. It does not see the next close. The target is scaled with the same training-fitted scaler and predictions are converted back to the original price unit before scoring.\n\nThe small regularization grid below is selected on validation MAE. Do not expand it after viewing test scores.\n"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": "X20 = windows[20].reshape(len(targets), 20)\nridge_candidates = {}\nridge_rows = []\nfor alpha in RIDGE_ALPHAS:\n    candidate = Ridge(alpha=alpha).fit(X20[train_mask], y_scaled[train_mask])\n    pred = original_units(candidate.predict(X20[valid_mask]))\n    ridge_candidates[alpha] = candidate\n    ridge_rows.append({'alpha':alpha, **scores(y_original[valid_mask], pred)})\nridge_validation = pd.DataFrame(ridge_rows).sort_values(['MAE','alpha'])\nselected_alpha = float(ridge_validation.iloc[0]['alpha'])\nridge_model = ridge_candidates[selected_alpha]\nridge_validation.to_csv(WORK / 'ridge_validation.csv', index=False)\nprint(ridge_validation.to_string(index=False))\nprint('Selected Ridge alpha:', selected_alpha)\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## 7. Train small LSTMs: lookbacks 10 versus 20\nThe model reads a sequence and returns one scaled price prediction. The LSTM state is reset between independent windows (stateful=False); we do not carry state across test examples. Early stopping watches **validation loss**, never test loss.\n\nWe reset the seed for each candidate, keep the architecture/optimizer/epoch budget fixed, and compare the same target dates. Changing the lookback changes the information window, not the number of features. The result is a sensitivity experiment, not a causal explanation.\n\n**Task C:** Before training, predict which lookback will work better and explain why. Afterwards, compare that expectation with the measured validation MAE. A prediction that turns out wrong is acceptable if you explain it honestly.\n"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": "lstm_candidates, histories, lstm_rows = {}, {}, []\nfor L in LOOKBACKS:\n    tf.keras.backend.clear_session()\n    tf.keras.utils.set_random_seed(SEED)\n    model = tf.keras.Sequential([\n        tf.keras.Input(shape=(L,1)),\n        tf.keras.layers.LSTM(HIDDEN_UNITS, stateful=False),\n        tf.keras.layers.Dense(1)\n    ])\n    model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=LEARNING_RATE), loss='mse')\n    history = model.fit(windows[L][train_mask], y_scaled[train_mask],\n                        validation_data=(windows[L][valid_mask], y_scaled[valid_mask]),\n                        epochs=EPOCHS, batch_size=BATCH_SIZE, shuffle=False, verbose=0,\n                        callbacks=[tf.keras.callbacks.EarlyStopping(\n                            monitor='val_loss', patience=3, restore_best_weights=True)])\n    pred = original_units(model.predict(windows[L][valid_mask], verbose=0))\n    lstm_candidates[L], histories[L] = model, history.history\n    lstm_rows.append({'lookback':L, 'epochs_run':len(history.history['loss']),\n                      **scores(y_original[valid_mask], pred)})\n    print('Finished lookback', L, lstm_rows[-1])\n\nlstm_validation = pd.DataFrame(lstm_rows).sort_values(['MAE','lookback'])\nselected_L = int(lstm_validation.iloc[0]['lookback'])\nlstm_model = lstm_candidates[selected_L]\nlstm_validation.to_csv(WORK / 'lstm_validation.csv', index=False)\nprint(lstm_validation.to_string(index=False))\nprint('Selected LSTM lookback:', selected_L)\nlstm_model.summary()\nfig, axes = plt.subplots(1,2, figsize=(12,4))\nfor ax, L in zip(axes, LOOKBACKS):\n    h = histories[L]\n    epoch_numbers = np.arange(1, len(h['loss'])+1)\n    ax.plot(epoch_numbers, h['loss'], label='Training')\n    ax.plot(epoch_numbers, h['val_loss'], label='Validation')\n    ax.set(title=f'Lookback {L}', xlabel='Epoch', ylabel='MSE (standardized price units)')\n    ax.legend()\nfig.tight_layout(); fig.savefig(FIGURES / '02_learning_curves.png'); plt.show()\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## 8. Freeze your decisions before opening the test gate\n**Stop and write:** use only the validation evidence above to explain your selected lookback and settings. Fill ANSWERS['validation_choice']. Do not claim the longer window must be better.\n\nThe following cell records the selected configuration. It does not retrain using validation data. Training-only fitted models and scaler remain fixed. Inspect your method, not the held-out answers, before continuing.\n"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": "selection = {'ticker':TICKER, 'seed':SEED, 'selected_lookback':selected_L,\n             'hidden_units':HIDDEN_UNITS, 'max_epochs':EPOCHS, 'batch_size':BATCH_SIZE,\n             'learning_rate':LEARNING_RATE, 'loss':'mse', 'optimizer':'Adam',\n             'early_stopping':{'monitor':'val_loss','patience':3,'restore_best_weights':True},\n             'ridge_alpha':selected_alpha, 'selection_metric':'validation MAE in original price units',\n             'lookbacks_compared':list(LOOKBACKS), 'ridge_alphas_compared':list(RIDGE_ALPHAS),\n             'snapshot_sha256':actual_hash, 'split_dates':split_record}\n(WORK / 'selection.json').write_text(json.dumps(selection, indent=2))\nprint(json.dumps(selection, indent=2))\n\nFINALIZE_TEST = False  # STUDENT: change to True only after documenting and freezing decisions.\npredictions = None\nif not FINALIZE_TEST:\n    print('Test gate CLOSED. Complete your validation explanation, then intentionally reopen this cell.')\nelse:\n    if not ANSWERS['validation_choice'].strip():\n        raise ValueError('Write your validation-based selection explanation in ANSWERS first.')\n    test_targets = targets[test_mask]\n    predictions = pd.DataFrame({\n        'target_date':df.loc[test_targets,'date'].to_numpy(),\n        'previous_close':prices[test_targets-1],\n        'observed_close':prices[test_targets],\n        'persistence_prediction':prices[test_targets-1],\n        'ridge_prediction':original_units(ridge_model.predict(X20[test_mask])),\n        'lstm_prediction':original_units(lstm_model.predict(windows[selected_L][test_mask], verbose=0))\n    })\n    metric_rows = [{'model':name, **scores(predictions['observed_close'], predictions[column])}\n                   for name,column in [('Persistence','persistence_prediction'),\n                                       ('Ridge','ridge_prediction'),('LSTM','lstm_prediction')]]\n    metrics = pd.DataFrame(metric_rows)\n    predictions.to_csv(WORK / 'test_predictions.csv', index=False)\n    metrics.to_csv(WORK / 'test_metrics.csv', index=False)\n    (WORK / 'test_evaluation.json').write_text(json.dumps({\n        'evaluated_utc':datetime.now(timezone.utc).isoformat(),\n        'note':'Do not tune settings after consulting these held-out results.',\n        'selection':selection}, indent=2))\n    print(metrics.to_string(index=False))\n    print('All models use',len(predictions),'identical target dates; units:',CURRENCY)\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## 9. Inspect aligned predictions and one failure case\nAfter the test gate is intentionally opened, examine the full held-out interval and a zoomed view. Identify the largest absolute LSTM error, and compare it with persistence on the **same date**.\n\n**Task D:** Is the LSTM better on MAE and RMSE? Does that justify a statement about trading profit or the direction of tomorrow's movement? Explain why not. Discuss underfitting/overfitting and the limits of a single instrument and a single test period. Do not invent news explanations for an error.\n"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": "if predictions is None:\n    print('Complete the validation explanation and open the test gate before this section.')\nelse:\n    fig, axes = plt.subplots(2,1,figsize=(12,8))\n    for ax, view, title in [(axes[0],predictions,'All test targets'),\n                            (axes[1],predictions.tail(60),'Last 60 test targets')]:\n        ax.plot(view['target_date'],view['observed_close'],label='Observed',color='black',linewidth=2)\n        for column,label in [('persistence_prediction','Persistence'),('ridge_prediction','Ridge'),\n                             ('lstm_prediction','LSTM')]:\n            ax.plot(view['target_date'],view[column],label=label,alpha=.8)\n        ax.set(title=title,ylabel=f'Price ({CURRENCY})'); ax.legend()\n    fig.tight_layout(); fig.savefig(FIGURES / '03_test_forecasts.png'); plt.show()\n    worst_index = (predictions['lstm_prediction']-predictions['observed_close']).abs().idxmax()\n    failure = predictions.loc[worst_index].copy()\n    print('Largest absolute LSTM test error:')\n    print(failure.to_string())\n    print('Absolute LSTM error:',abs(failure['lstm_prediction']-failure['observed_close']))\n    print('Absolute persistence error:',abs(failure['persistence_prediction']-failure['observed_close']))\n    failure.to_csv(WORK / 'largest_error_case.csv', header=['value'])\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## 10. Demonstrate the prediction system on an actual held-out window\nThe function below takes the row number of a test-period target and returns the available input dates, the three forecasts, and the later-observed target. It does **not** use the target price to create forecasts. Displaying the observed target afterwards is evaluation, not an input feature.\n\nFor your video, show an input window and describe the complete path from CSV to a prediction. This is a historical held-out demonstration, not live market execution.\n"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": "def forecast_for_test_target(target_row):\n    if predictions is None:\n        raise RuntimeError('The final test gate is still closed.')\n    j = int(target_row)\n    if not valid_end <= j < n:\n        raise ValueError('Choose a target row inside the held-out test period.')\n    x_lstm = scaled[j-selected_L:j].reshape(1,selected_L,1)\n    x_ridge = scaled[j-20:j].reshape(1,20)\n    forecast = {'target_date':str(df.loc[j,'date'].date()),\n                'input_start':str(df.loc[j-selected_L,'date'].date()),\n                'input_end':str(df.loc[j-1,'date'].date()),\n                'persistence_prediction':float(prices[j-1]),\n                'ridge_prediction':float(original_units(ridge_model.predict(x_ridge))[0]),\n                'lstm_prediction':float(original_units(lstm_model.predict(x_lstm,verbose=0))[0])}\n    print('Forecasts using past observations only:',json.dumps(forecast,indent=2))\n    print('Observed target, revealed for evaluation:',float(prices[j]))\n    return forecast\n\nif predictions is not None:\n    demo = forecast_for_test_target(valid_end)\nelse:\n    print('The demonstration becomes available after you intentionally evaluate the final test set.')\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## 11. Explain, disclose, and package your work\nFill all entries in ANSWERS with your own evidence-based explanations. Include the actual AI tool/model, representative prompts, what you changed, and what you personally checked. Write 'No AI assistance' if appropriate. Do not paste secrets into your notebook or disclosure.\n\nExplain what the LSTM is intended to remember, and why the lookback experiment does not identify a causal driver of stock prices. No attention map or attribution chart is required.\n\n**Important:** the export below packages your current files; it does not judge whether your explanations are correct. Empty answers or a closed test gate produce a DRAFT package. Include the separately downloaded **executed notebook** in your final submission. The archive can contain licensed market data: share it only through permitted course submission channels.\n"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": "# STUDENT: update ANSWERS in section 1, then run this export after recording your video.\nmissing_answers = [key for key,value in ANSWERS.items() if not str(value).strip()]\nstatus = 'draft' if predictions is None or missing_answers else 'student-completed; not automatically validated'\n(WORK / 'student_explanations.json').write_text(json.dumps(ANSWERS,indent=2))\n(WORK / 'submission_status.json').write_text(json.dumps({\n    'status':status, 'missing_answers':missing_answers,\n    'test_evaluated_this_run':predictions is not None,\n    'reminder':'Add the executed .ipynb and your video. Check data redistribution permissions.'},indent=2))\njoblib.dump({'ridge':ridge_model, 'scaler':scaler, 'selection':selection}, WORK / 'ridge_and_preprocessing.joblib')\nlstm_model.save(WORK / 'lstm.keras')\n(WORK / 'README.txt').write_text(\n    'Stock forecasting HW - Machine Learning in Engineering Science, Fall 2026\\n'\n    '1. Include your executed stock_forecasting.ipynb with this package.\\n'\n    '2. Install requirements.txt in an appropriate Python environment.\\n'\n    '3. Preserve the CSV and provenance; use the original working-folder layout.\\n'\n    '4. Run the notebook in order; inspect selection.json and split_dates.json.\\n'\n    '5. Model selection uses validation only; final test results are not tuning feedback.\\n'\n    '6. See student_explanations.json for interpretation, AI use, and video location.\\n'\n    '7. Check dataset sharing permissions before distributing this archive.\\n'\n    'Status: '+status+'\\n')\narchive = shutil.make_archive('stock_forecasting_submission','zip',WORK)\nprint('Package:',archive,'| Status:',status)\nprint('Missing answers:',missing_answers)\nprint('Download the executed notebook separately and add it to your final submission.')\n# Colab: uncomment the following two lines when ready to download the archive.\n# from google.colab import files\n# files.download(archive)\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## 12. Your five-minute English video\n| Time | Show and explain |\n|---|---|\n| 0:00-0:30 | Instrument, forecast horizon, available information, and research question. |\n| 0:30-1:20 | Data provenance, split boundaries, one input/target example, and leakage prevention. |\n| 1:20-2:20 | Persistence, Ridge, LSTM input shape, architecture, and validation-based lookback choice. |\n| 2:20-3:35 | A historical held-out system demonstration, score table, and aligned forecast plots. |\n| 3:35-4:30 | Largest error, whether LSTM beat persistence, and what the result cannot establish. |\n| 4:30-5:00 | Reproduction instructions, AI-use disclosure, and what you personally learned. |\n\nUse your own narration. Aim for 5:00 (acceptable: 4:30-5:30). Show actual notebook results; do not spend the recording waiting for training. Webcam optional. Confirm your instructor can access the video.\n\n### Final checklist\n- [ ] All three models evaluated on identical held-out target dates, in original units.\n- [ ] Training-only preprocessing; no test-based model selection.\n- [ ] Input/target dates and array shapes explained.\n- [ ] Lookback comparison, learning curves, aligned forecasts, and failure analysis included.\n- [ ] Dataset fingerprint, provenance, environment, seed, and settings saved.\n- [ ] All student explanations and the AI-use log completed honestly.\n- [ ] Executed notebook, permitted data/reproduction route, exported results, and video submitted.\n\n### Official technical references\n- [TensorFlow time-series tutorial](https://www.tensorflow.org/tutorials/structured_data/time_series)\n- [scikit-learn leakage pitfalls](https://scikit-learn.org/stable/common_pitfalls.html)\n- [Ridge API](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html)\n- [Keras LSTM](https://keras.io/api/layers/recurrent_layers/lstm/)\n- [Keras EarlyStopping](https://keras.io/api/callbacks/early_stopping/)\n- [yfinance download API](https://ranaroussi.github.io/yfinance/reference/api/yfinance.download.html)\n\nEducational forecasting only; no investment or profitability claim is supported merely by a price-prediction chart.\n"
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "colab": {
      "name": "stock-market-prediction-starter.ipynb",
      "provenance": []
    },
    "course": {
      "title": "Machine Learning in Engineering Science",
      "semester": "Fall 2026",
      "type": "homework starter",
      "execution_status": "Not executed before publication",
      "seed": 2026
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}