{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "52fa8aed",
   "metadata": {},
   "source": [
    "# Lab 2: A Neural Network with Multiple Layers\n",
    "\n",
    "**Special Topics in Artificial Intelligence (STAI 2026)**\n",
    "\n",
    "In Lab 1, one node learned a straight-line relationship and beat the baseline. In this lab you build a **multilayer perceptron (MLP)**: the same inputs, but now with **hidden layers** and an **activation function**, so the network can learn curved relationships. You will train it, evaluate it, compare it fairly against Lab 1, and use it for inference.\n",
    "\n",
    "**How to run:** in Google Colab, choose *Runtime > Run all*. No GPU is needed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8878505",
   "metadata": {},
   "source": [
    "## 1) Setup, data, split, scale, baseline\n",
    "\n",
    "These steps are identical to Lab 1, so they are condensed here: load `Concrete Compressive Strength.csv` (in Google Colab, the cell opens an upload dialog: choose the CSV from your computer), split 80/20, standardize using statistics from the training data only, and compute the mean baseline.\n",
    "\n",
    "**Dataset citation:** Yeh, I. (1998). Concrete Compressive Strength [Dataset]. UCI Machine Learning Repository. https://doi.org/10.24432/C5PK67."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "005e9c98",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import matplotlib.pyplot as plt\n",
    "from pathlib import Path\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "torch.manual_seed(0)\n",
    "np.random.seed(0)\n",
    "\n",
    "CSV_NAME = 'Concrete Compressive Strength.csv'\n",
    "if not Path(CSV_NAME).exists():\n",
    "    from google.colab import files     # Colab upload dialog\n",
    "    files.upload()\n",
    "\n",
    "df = pd.read_csv(CSV_NAME)\n",
    "df.columns = df.columns.str.strip()\n",
    "TARGET = 'Concrete compressive strength'\n",
    "X_df = df.drop(columns=[TARGET])\n",
    "X = X_df.values.astype(np.float32)\n",
    "y = df[[TARGET]].values.astype(np.float32)\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.2, random_state=42)\n",
    "\n",
    "scaler = StandardScaler().fit(X_train)     # training data only\n",
    "X_train_s = scaler.transform(X_train)\n",
    "X_test_s  = scaler.transform(X_test)\n",
    "\n",
    "X_train_t = torch.tensor(X_train_s)\n",
    "y_train_t = torch.tensor(y_train)\n",
    "X_test_t  = torch.tensor(X_test_s)\n",
    "\n",
    "baseline_mae = np.abs(y_test - y_train.mean()).mean()\n",
    "print('Rows:', X.shape[0], '| Baseline test MAE:', round(float(baseline_mae), 2), 'MPa')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47e64669",
   "metadata": {},
   "source": [
    "## 2) From one node to hidden layers\n",
    "\n",
    "Our network now has this shape:\n",
    "\n",
    "**8 inputs → 32 nodes → 16 nodes → 1 output**\n",
    "\n",
    "The two middle layers are **hidden layers**. After each hidden layer we apply **ReLU**, an activation function that outputs the input itself when it is positive, and 0 otherwise. The activation is what makes depth useful: without it, a stack of linear layers collapses into one big linear layer, no better than Lab 1.\n",
    "\n",
    "**Count the parameters, layer by layer** (the same calculation as in the quiz):\n",
    "\n",
    "- input to hidden 1: $8 \\times 32$ weights $+ 32$ biases $= 288$\n",
    "- hidden 1 to hidden 2: $32 \\times 16$ weights $+ 16$ biases $= 528$\n",
    "- hidden 2 to output: $16 \\times 1$ weights $+ 1$ bias $= 17$\n",
    "\n",
    "Total: $288 + 528 + 17 = 833$ parameters, compared with 9 in Lab 1. The code below confirms this count."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2bfed46",
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_mlp():\n",
    "    return nn.Sequential(\n",
    "        nn.Linear(8, 32),   # input -> hidden layer 1\n",
    "        nn.ReLU(),\n",
    "        nn.Linear(32, 16),  # hidden layer 1 -> hidden layer 2\n",
    "        nn.ReLU(),\n",
    "        nn.Linear(16, 1),   # hidden layer 2 -> output\n",
    "    )\n",
    "\n",
    "mlp = make_mlp()\n",
    "print(mlp)\n",
    "print('Total parameters:', sum(p.numel() for p in mlp.parameters()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89363eff",
   "metadata": {},
   "source": [
    "## 3) Training\n",
    "\n",
    "The training loop is exactly the same four steps as in Lab 1: forward, loss, backward, update. Only the model is bigger. We wrap the loop in a function so we can reuse it for a fair comparison later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ca3086e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(model, epochs=2000, lr=0.01, report_every=400):\n",
    "    loss_fn   = nn.MSELoss()\n",
    "    optimizer = torch.optim.Adam(model.parameters(), lr=lr)\n",
    "    losses = []\n",
    "    for epoch in range(epochs):\n",
    "        optimizer.zero_grad()\n",
    "        loss = loss_fn(model(X_train_t), y_train_t)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        losses.append(loss.item())\n",
    "        if (epoch + 1) % report_every == 0:\n",
    "            print(f'Epoch {epoch+1:4d} | training MSE: {loss.item():8.2f}')\n",
    "    return losses\n",
    "\n",
    "losses = train(mlp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5322e768",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(7, 4))\n",
    "plt.plot(losses)\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Training MSE')\n",
    "plt.title('MLP training loss')\n",
    "plt.grid(alpha=0.3)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "669fb198",
   "metadata": {},
   "source": [
    "## 4) Evaluation and a fair comparison\n",
    "\n",
    "We evaluate on the test set, and we also train a fresh one-node model (Lab 1) with the same function, so the comparison uses identical data and settings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0253adf",
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_mae(model):\n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        pred = model(X_test_t).numpy()\n",
    "    return np.abs(y_test - pred).mean(), pred\n",
    "\n",
    "one_node = nn.Sequential(nn.Linear(8, 1))\n",
    "train(one_node, epochs=2000, lr=0.05, report_every=2000)\n",
    "\n",
    "mae_mlp, pred_mlp = test_mae(mlp)\n",
    "mae_one, _        = test_mae(one_node)\n",
    "\n",
    "print()\n",
    "print(f'Baseline (predict the mean) : MAE = {baseline_mae:5.2f} MPa')\n",
    "print(f'One node (Lab 1)            : MAE = {mae_one:5.2f} MPa')\n",
    "print(f'MLP with 2 hidden layers    : MAE = {mae_mlp:5.2f} MPa')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c21a07b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(5.5, 5.5))\n",
    "plt.scatter(y_test, pred_mlp, s=14, alpha=0.6, color='teal')\n",
    "lims = [0, 85]\n",
    "plt.plot(lims, lims, 'r--', lw=1, label='perfect prediction')\n",
    "plt.xlabel('Actual strength (MPa)')\n",
    "plt.ylabel('Predicted strength (MPa)')\n",
    "plt.title('MLP: predicted versus actual')\n",
    "plt.legend(); plt.grid(alpha=0.3)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e5ca956",
   "metadata": {},
   "source": [
    "Compare this scatter plot with the one from Lab 1: the points sit much closer to the red line, including at the low and high ends. The hidden layers let the network bend its predictions to follow the real, curved behavior of concrete.\n",
    "\n",
    "## 5) Inference: the same new mix as in Lab 1\n",
    "\n",
    "Same rule as before: scale the new input with the scaler fitted on the training data, then predict."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d4345ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "new_mix = pd.DataFrame([{\n",
    "    'Cement': 320.0,\n",
    "    'Blast Furnace Slag': 0.0,\n",
    "    'Fly Ash': 100.0,\n",
    "    'Water': 180.0,\n",
    "    'Superplasticizer': 8.0,\n",
    "    'Coarse Aggregate': 970.0,\n",
    "    'Fine Aggregate': 780.0,\n",
    "    'Age (day)': 28,\n",
    "}])\n",
    "\n",
    "new_mix = new_mix[X_df.columns]            # same column order as training\n",
    "new_scaled = scaler.transform(new_mix.values.astype(np.float32))\n",
    "with torch.no_grad():\n",
    "    predicted = mlp(torch.tensor(new_scaled)).item()\n",
    "\n",
    "print(f'MLP prediction for the new mix at 28 days: {predicted:.1f} MPa')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afdf09bf",
   "metadata": {},
   "source": [
    "## 6) What you learned\n",
    "\n",
    "- Hidden layers plus an activation function let a network learn curved relationships; without the activation, depth adds nothing.\n",
    "- The MLP uses 833 parameters instead of 9, and rewards us with a clearly lower test error than the one-node model and the baseline.\n",
    "- Training and inference follow exactly the same recipe as in Lab 1; only the model changed.\n",
    "\n",
    "### Exercises\n",
    "\n",
    "1. Remove both `nn.ReLU()` lines from `make_mlp` and retrain. Compare the test MAE with the one-node model. What does this confirm about activation functions?\n",
    "2. Make the network wider (for example 64 and 32 nodes) or deeper (a third hidden layer). Does the test MAE improve, and what happens to the parameter count?\n",
    "3. Train for 8,000 epochs and watch the training loss and the test MAE. Do they keep improving together? This prepares the topic of overfitting.\n",
    "4. Compute the MLP's prediction for the Lab 1 exercise mixes (Age 3 and Age 90). Are the predictions more realistic than the one-node ones?"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}