{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1396f392",
   "metadata": {},
   "source": [
    "# Lab 3: Classification with a Multilayer Perceptron\n",
    "\n",
    "**Special Topics in Artificial Intelligence (STAI 2026)**\n",
    "\n",
    "In Labs 1 and 2 the network predicted a **number** (concrete strength in MPa): that was **regression**. In this lab the network answers a **yes or no question**: is a tumor malignant or benign? That is **binary classification**, and three things change:\n",
    "\n",
    "1. the **output** of the network is turned into a probability between 0 and 1,\n",
    "2. the **loss function** changes from mean squared error to **binary cross-entropy**,\n",
    "3. the **evaluation** changes: accuracy alone is not enough, and we must decide which mistakes matter most.\n",
    "\n",
    "You will build the same kind of multilayer perceptron (MLP) as in Lab 2, train it, evaluate it with a confusion matrix, and tune the decision threshold to trade one kind of mistake against another.\n",
    "\n",
    "**How to run:** in Google Colab, choose *Runtime > Run all*. No GPU is needed; everything runs on the CPU in under a minute. Read the text between the cells: it explains what each step does and why."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d0f3c5b",
   "metadata": {},
   "source": [
    "## 1) Setup\n",
    "\n",
    "Everything we need is already installed in Google Colab, so there is nothing to install. The random seeds make the results repeatable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3e53ddb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2026-11-03T20:05:31.582694Z",
     "start_time": "2026-11-03T20:05:28.284870Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import matplotlib.pyplot as plt\n",
    "from pathlib import Path\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "torch.manual_seed(0)\n",
    "np.random.seed(0)\n",
    "print('PyTorch version:', torch.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f35252b7",
   "metadata": {},
   "source": [
    "## 2) The dataset: breast cancer diagnosis\n",
    "\n",
    "We use the **Breast Cancer Wisconsin (Diagnostic)** dataset (file: `breast_cancer.csv`). Each of the **569 rows** is one tumor sample. The **30 numeric input columns** are measurements computed from a microscope image of cell nuclei: the mean, standard error, and worst value of properties such as radius, texture, perimeter, area, and smoothness.\n",
    "\n",
    "The **target column** is `diagnosis`: **M** means malignant (cancer) and **B** means benign (not cancer). The dataset contains **212 malignant** and **357 benign** samples.\n",
    "\n",
    "This is a **binary classification** task: the answer is one of two classes, not a number. All inputs are numeric, so no one-hot encoding is needed.\n",
    "\n",
    "**Loading the file:** the next cell looks for the CSV next to the notebook. In Google Colab the file is not there yet, so the cell opens an **upload dialog**: choose `breast_cancer.csv` from your computer, wait until the upload finishes, and the data loads automatically.\n",
    "\n",
    "**Dataset citation:** Wolberg, W., Mangasarian, O., Street, N., and Street, W. (1993). Breast Cancer Wisconsin (Diagnostic) [Dataset]. UCI Machine Learning Repository. https://doi.org/10.24432/C5DW2B."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3291438c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2026-11-03T20:05:31.623019Z",
     "start_time": "2026-11-03T20:05:31.585361Z"
    }
   },
   "outputs": [],
   "source": [
    "CSV_NAME = 'breast_cancer.csv'\n",
    "\n",
    "if not Path(CSV_NAME).exists():\n",
    "    # Running in Google Colab: open the upload dialog and choose the CSV file.\n",
    "    from google.colab import files\n",
    "    files.upload()\n",
    "\n",
    "df = pd.read_csv(CSV_NAME)\n",
    "print(df.shape)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "995aa53e",
   "metadata": {},
   "source": [
    "## 3) Prepare the target: from letters to numbers\n",
    "\n",
    "A network cannot work with the letters M and B, so we turn the diagnosis into a number: **malignant becomes 1** and **benign becomes 0**. The code below does this with a small translation table, `{'M': 1, 'B': 0}`, applied to the diagnosis column.\n",
    "\n",
    "The choice of which class gets the 1 matters, because the 1 is the **positive class**: the thing our detector is searching for. Here we search for cancer, so malignant is the positive class. Every metric later in this lab (precision, recall) is defined around this choice.\n",
    "\n",
    "We also drop the `id` column: a patient number carries no medical information, and a model that uses it would learn nothing useful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "029c4312",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2026-11-03T20:05:31.632084Z",
     "start_time": "2026-11-03T20:05:31.625075Z"
    }
   },
   "outputs": [],
   "source": [
    "# Step 1: translate the letters into numbers, M -> 1 and B -> 0\n",
    "df['diagnosis'] = df['diagnosis'].map({'M': 1, 'B': 0})\n",
    "\n",
    "# Step 2: the inputs are the 30 measurement columns (no id, no diagnosis, no target)\n",
    "X_df = df.drop(columns=['id', 'diagnosis'])\n",
    "\n",
    "# Step 3: the labels, as one column of 0s and 1s.\n",
    "# The double brackets [['target']] keep y as a one-column table,\n",
    "# which matches the one-column output of the network.\n",
    "y = df[['diagnosis']].values.astype(np.float32)\n",
    "\n",
    "print('Inputs:', X_df.shape, '| Labels:', y.shape)\n",
    "print('Malignant (y=1):', int((y == 1).sum()), '| Benign (y=0):', int((y == 0).sum()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8641c62",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2026-11-03T20:05:31.682409Z",
     "start_time": "2026-11-03T20:05:31.634806Z"
    }
   },
   "outputs": [],
   "source": [
    "display(df.head())\n",
    "display(df.tail())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eaf597b6",
   "metadata": {},
   "source": [
    "## 4) Split the data, keeping the class balance\n",
    "\n",
    "As always: train on one part, test on data the model has never seen. For classification there is one extra care: the split should keep the **same proportion of malignant and benign** in both parts. Otherwise the test set could, by bad luck, contain mostly one class, and the evaluation would be distorted. `stratify=y` does exactly this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03dd5b67",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2026-11-03T20:05:31.693898Z",
     "start_time": "2026-11-03T20:05:31.684441Z"
    }
   },
   "outputs": [],
   "source": [
    "X = X_df.values.astype(np.float32)\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.2, random_state=42, stratify=y)\n",
    "\n",
    "print('Training rows:', X_train.shape[0], '| malignant:', int((y_train == 1).sum()))\n",
    "print('Test rows:    ', X_test.shape[0], '| malignant:', int((y_test == 1).sum()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa390c8f",
   "metadata": {},
   "source": [
    "## 5) Scale the features\n",
    "\n",
    "Same rule as in Labs 1 and 2: the 30 columns have very different ranges (area is in the hundreds, smoothness is around 0.1), so we standardize them, and the scaler is fitted on the **training data only**. Fitting it on all data would leak information from the test set into training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0fefecf3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2026-11-03T20:05:31.704096Z",
     "start_time": "2026-11-03T20:05:31.695798Z"
    }
   },
   "outputs": [],
   "source": [
    "scaler = StandardScaler()\n",
    "scaler.fit(X_train)                        # statistics come from training data only\n",
    "\n",
    "X_train_s = scaler.transform(X_train)\n",
    "X_test_s  = scaler.transform(X_test)\n",
    "\n",
    "print('Training mean per column (after scaling), first 5:', X_train_s.mean(axis=0)[:5].round(2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55f4823a",
   "metadata": {},
   "source": [
    "## 6) A baseline, and a first warning about accuracy\n",
    "\n",
    "The simplest possible classifier always predicts the **majority class**. Benign is the majority, so this baseline says \"benign\" for every patient.\n",
    "\n",
    "Look carefully at the two numbers the next cell prints. Without looking at a single measurement, the baseline already gets almost two of every three patients right, simply because most samples are benign. And in tasks where 99% of samples belong to one class (the Week 4 anomaly example), this same trick produces 99% accuracy. Yet it finds **zero** of the malignant tumors: for a cancer detector, complete failure. Remember this: in classification, **accuracy alone can hide exactly the mistakes we care about most**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "070a3ece",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2026-11-03T20:05:31.714431Z",
     "start_time": "2026-11-03T20:05:31.705991Z"
    }
   },
   "outputs": [],
   "source": [
    "# The baseline always answers 'benign', so its prediction is 0 for every patient.\n",
    "baseline_prediction = np.zeros_like(y_test)     # an array of zeros with the same shape as y_test\n",
    "\n",
    "correct = (baseline_prediction == y_test).sum() # count how many of its answers are correct\n",
    "baseline_accuracy = correct / len(y_test)\n",
    "\n",
    "malignant_in_test = int((y_test == 1).sum())\n",
    "\n",
    "print(f'Baseline accuracy:        {baseline_accuracy:.1%}')\n",
    "print(f'Malignant tumors found:   0 of {malignant_in_test} (this baseline never says malignant)')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af132e2a",
   "metadata": {},
   "source": [
    "## 7) The model: an MLP with a probability output\n",
    "\n",
    "The network is the same kind of architecture as in Lab 2 (two hidden layers with ReLU), now with 30 inputs and smaller hidden layers:\n",
    "\n",
    "**30 inputs → 16 nodes → 8 nodes → 1 output**\n",
    "\n",
    "**Count the parameters, layer by layer:** $30 \\times 16 + 16 = 496$, then $16 \\times 8 + 8 = 136$, then $8 \\times 1 + 1 = 9$. Total: **641 parameters**.\n",
    "\n",
    "### Why do we need the sigmoid?\n",
    "\n",
    "Think about what the output node actually computes: a weighted sum plus a bias, exactly as in Lab 1. A weighted sum can be **any number**: 0.3, or 7.9, or minus 250. This raw output is called the **logit**. You can read the logit as the strength of the evidence the network has collected: strongly positive means strong evidence for malignant, strongly negative means strong evidence for benign.\n",
    "\n",
    "But our question needs an answer we can act on: **what is the probability that this tumor is malignant?** A probability must live between 0 and 1, and \"7.9\" or \"minus 250\" are not probabilities. So we need a translator from \"any number\" to \"a number between 0 and 1\". That translator is the **sigmoid** function:\n",
    "\n",
    "$$p = \\sigma(z) = \\frac{1}{1 + e^{-z}}$$\n",
    "\n",
    "Here $e \\approx 2.718$ is a fixed mathematical constant; its exact value does not matter for us, only the squashing shape it produces. The next cell draws that shape. Check these landmark values in the plot:\n",
    "\n",
    "| logit $z$ | $-4$ | $-1$ | $0$ | $+1$ | $+4$ |\n",
    "|:---:|:---:|:---:|:---:|:---:|:---:|\n",
    "| probability $\\sigma(z)$ | 0.02 | 0.27 | 0.50 | 0.73 | 0.98 |\n",
    "\n",
    "A logit of 0 means \"no evidence either way\": probability exactly 0.5, completely unsure. The further the logit moves from 0, the closer the probability moves to 0 or 1, but it never leaves that range. That is exactly the behavior a probability needs.\n",
    "\n",
    "In our model, the sigmoid is simply the **last step of the network**: first the layers compute the logit, then the sigmoid translates it into a probability. So the model's output is already the probability of malignant, ready to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfb7b70d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2026-11-03T20:05:31.942918Z",
     "start_time": "2026-11-03T20:05:31.716334Z"
    }
   },
   "outputs": [],
   "source": [
    "z = np.linspace(-8, 8, 200)\n",
    "sigmoid = 1 / (1 + np.exp(-z))\n",
    "\n",
    "plt.figure(figsize=(7, 4))\n",
    "plt.plot(z, sigmoid, lw=2)\n",
    "plt.axhline(0.5, color='gray', ls='--', lw=1)\n",
    "plt.axvline(0, color='gray', ls='--', lw=1)\n",
    "plt.scatter([0], [0.5], color='red', zorder=3, label='logit 0 -> probability 0.5')\n",
    "plt.xlabel('logit z (any number)')\n",
    "plt.ylabel('sigmoid(z) = probability (0 to 1)')\n",
    "plt.title('The sigmoid translates any number into a probability')\n",
    "plt.legend(); plt.grid(alpha=0.3)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84866b4c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2026-11-03T20:05:31.953080Z",
     "start_time": "2026-11-03T20:05:31.944823Z"
    }
   },
   "outputs": [],
   "source": [
    "model = nn.Sequential(\n",
    "    nn.Linear(30, 16),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(16, 8),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(8, 1),     # this layer computes the logit\n",
    "    nn.Sigmoid(),        # last step: translate the logit into a probability (0 to 1)\n",
    ")\n",
    "\n",
    "n_params = 0\n",
    "for p in model.parameters():\n",
    "    n_params = n_params + p.numel()   # numel counts the numbers in each block of weights or biases\n",
    "\n",
    "print(model)\n",
    "print('Total parameters:', n_params)   # 496 + 136 + 9 = 641"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "faeb6bcb",
   "metadata": {},
   "source": [
    "## 8) The loss function for binary classification\n",
    "\n",
    "### Why not mean squared error?\n",
    "\n",
    "In Labs 1 and 2 the loss was the mean squared error: the distance between a predicted number and the true number. Here the model predicts a **probability**, and the true answer is a **class** (malignant or benign). We need a loss that asks a different question: **how much probability did the model give to the correct answer?** A model that gives the correct class a probability of 0.9 should receive a small loss; a model that gives it only 0.1 should receive a large one.\n",
    "\n",
    "### Binary cross-entropy, one case at a time\n",
    "\n",
    "That loss is **binary cross-entropy (BCE)**. For one sample with true label $y$ (1 for malignant, 0 for benign) and predicted probability $p$:\n",
    "\n",
    "$$\\text{BCE} = -\\left[\\, y \\log(p) + (1 - y) \\log(1 - p) \\,\\right]$$\n",
    "\n",
    "The formula looks busy, but one of its two terms is always zero. One reading note first: $\\log$ here means the **natural logarithm** (the `ln` key on a calculator); checking the numbers below with the `log` key instead gives different values.\n",
    "\n",
    "- **Malignant sample ($y = 1$):** the second term vanishes, and the loss is $-\\log(p)$: the punishment for not giving probability to \"malignant\".\n",
    "- **Benign sample ($y = 0$):** the first term vanishes, and the loss is $-\\log(1 - p)$: the punishment for not giving probability to \"benign\".\n",
    "\n",
    "So in both cases the rule is the same: **take the probability the model gave to the correct answer, and punish with $-\\log$ of it.**\n",
    "\n",
    "### A worked example: one patient, three models\n",
    "\n",
    "A patient's tumor is malignant ($y = 1$). Three different models make a prediction:\n",
    "\n",
    "| Model says | Meaning | Loss $= -\\log(p_\\text{correct})$ | Verdict |\n",
    "|:---|:---|:---:|:---|\n",
    "| $p = 0.9$ | confident, correct | $-\\log(0.9) \\approx 0.11$ | small punishment |\n",
    "| $p = 0.5$ | completely unsure | $-\\log(0.5) \\approx 0.69$ | medium punishment |\n",
    "| $p = 0.1$ | confident, wrong | $-\\log(0.1) \\approx 2.30$ | heavy punishment |\n",
    "\n",
    "The confident wrong answer costs about **twenty times more** than the confident correct one. And the punishment keeps exploding: $p = 0.01$ would cost 4.61, and $p = 0.001$ would cost 6.91. This is the personality of cross-entropy: **a confident wrong answer is punished very heavily**, which pushes the model toward honest probabilities. The same table works for a benign patient; only the roles swap, because the loss is then $-\\log(1-p)$.\n",
    "\n",
    "The next cell computes this table, so you can see the numbers are real and try your own values of $p$.\n",
    "\n",
    "### The loss in the code\n",
    "\n",
    "In PyTorch this loss is `nn.BCELoss()`, read as \"binary cross-entropy loss\". It receives the **probability** that our model outputs (remember: the sigmoid is the last step of the model) and the true label, and computes exactly the formula above. The full path of one prediction is:\n",
    "\n",
    "$$\\text{layers} \\;\\xrightarrow{\\;}\\; \\text{logit} \\;\\xrightarrow{\\;\\text{sigmoid}\\;}\\; \\text{probability} \\;\\xrightarrow{\\;\\text{BCE}\\;}\\; \\text{loss}$$\n",
    "\n",
    "During training, the loss of a whole epoch is simply the **average BCE over all training samples**. The training loop itself is unchanged from Labs 1 and 2: forward, loss, backward, update.\n",
    "\n",
    "(A side note for later courses: PyTorch also offers `nn.BCEWithLogitsLoss`, a variant that takes the raw logit and applies the sigmoid inside the loss. It behaves the same but is numerically safer for very large networks. We use the visible version here because it is easier to read.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba3d2eaf",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2026-11-03T20:05:31.961962Z",
     "start_time": "2026-11-03T20:05:31.956122Z"
    }
   },
   "outputs": [],
   "source": [
    "# The worked example from the text: a malignant patient (y = 1), three models.\n",
    "for p in [0.9, 0.5, 0.1]:\n",
    "    loss = -np.log(p)                  # probability given to the correct answer: p\n",
    "    print(f'model says p = {p:3.1f} -> loss = -log({p:3.1f}) = {loss:.2f}')\n",
    "\n",
    "print()\n",
    "# The same three predictions for a benign patient (y = 0):\n",
    "for p in [0.9, 0.5, 0.1]:\n",
    "    loss = -np.log(1 - p)              # probability given to the correct answer: 1 - p\n",
    "    print(f'model says p = {p:3.1f} -> loss = -log(1 - {p:3.1f}) = {loss:.2f}  (benign patient)')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c85e0139",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2026-11-03T20:05:33.277699Z",
     "start_time": "2026-11-03T20:05:31.963891Z"
    }
   },
   "outputs": [],
   "source": [
    "X_train_t = torch.tensor(X_train_s)\n",
    "y_train_t = torch.tensor(y_train)\n",
    "X_test_t  = torch.tensor(X_test_s)\n",
    "\n",
    "loss_fn   = nn.BCELoss()                  # binary cross-entropy on probabilities\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=0.001)\n",
    "\n",
    "losses = []\n",
    "for epoch in range(200):\n",
    "    optimizer.zero_grad()                 # clear old gradients\n",
    "    predicted_probs = model(X_train_t)    # 1) forward: the model outputs probabilities\n",
    "    loss = loss_fn(predicted_probs, y_train_t)   # 2) loss: compare with the true labels\n",
    "    loss.backward()                       # 3) backward\n",
    "    optimizer.step()                      # 4) update\n",
    "    losses.append(loss.item())\n",
    "    if (epoch + 1) % 40 == 0:\n",
    "        print(f'Epoch {epoch+1:3d} | training BCE loss: {loss.item():.4f}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8fa42a94",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2026-11-03T20:05:33.466351Z",
     "start_time": "2026-11-03T20:05:33.280273Z"
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(7, 4))\n",
    "plt.plot(losses)\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Training BCE loss')\n",
    "plt.title('The loss falls as the network learns to classify')\n",
    "plt.grid(alpha=0.3)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d7dd318",
   "metadata": {},
   "source": [
    "## 9) Evaluation: the confusion matrix\n",
    "\n",
    "To evaluate, we compute the probability for every test sample and apply a **decision threshold**: if $p \\ge 0.5$, predict malignant; otherwise predict benign. Then we sort every test prediction into one of four boxes. In this medical task, each box has a very concrete meaning:\n",
    "\n",
    "- **True positive (TP):** malignant tumor, correctly flagged. A cancer that was caught.\n",
    "- **False negative (FN):** malignant tumor, predicted benign. **A missed cancer. This is the dangerous mistake.**\n",
    "- **False positive (FP):** benign tumor, flagged as malignant. A false alarm: the patient gets further examination that turns out unnecessary.\n",
    "- **True negative (TN):** benign tumor, correctly cleared.\n",
    "\n",
    "From these four counts come the three metrics you know from Week 4:\n",
    "\n",
    "- **Accuracy** $= (TP + TN) / \\text{all}$: the share of correct decisions.\n",
    "- **Precision** $= TP / (TP + FP)$: when we flag a tumor, how often is it really malignant?\n",
    "- **Recall** $= TP / (TP + FN)$: of all real cancers, how many did we catch?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff56bda0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2026-11-03T20:14:31.796182Z",
     "start_time": "2026-11-03T20:14:31.785888Z"
    }
   },
   "outputs": [],
   "source": [
    "model.eval()\n",
    "with torch.no_grad():\n",
    "    probabilities = model(X_test_t).numpy()   # the model outputs one probability per patient\n",
    "\n",
    "def evaluate(probs, y_true, threshold):\n",
    "    # Decision rule: flag as malignant (1) when the probability reaches the threshold.\n",
    "    # (probs >= threshold) gives True or False for each patient; astype turns that into 1.0 or 0.0.\n",
    "    predictions = (probs >= threshold).astype(np.float32)\n",
    "\n",
    "    # The four boxes of the confusion matrix.\n",
    "    # Reading example: (predictions == 1) & (y_true == 1) is True only when BOTH sides\n",
    "    # are true for that patient, and .sum() counts for how many patients that happens.\n",
    "    tp = int(((predictions == 1) & (y_true == 1)).sum())   # malignant and flagged: caught\n",
    "    fp = int(((predictions == 1) & (y_true == 0)).sum())   # benign but flagged: false alarm\n",
    "    fn = int(((predictions == 0) & (y_true == 1)).sum())   # malignant but not flagged: missed\n",
    "    tn = int(((predictions == 0) & (y_true == 0)).sum())   # benign and not flagged: cleared\n",
    "\n",
    "    accuracy = (tp + tn) / len(y_true)\n",
    "\n",
    "    if tp + fp > 0:\n",
    "        precision = tp / (tp + fp)\n",
    "    else:\n",
    "        precision = float('nan')      # the model raised no alerts, so precision is undefined\n",
    "\n",
    "    if tp + fn > 0:\n",
    "        recall = tp / (tp + fn)\n",
    "    else:\n",
    "        recall = float('nan')\n",
    "\n",
    "    return tp, fp, fn, tn, accuracy, precision, recall\n",
    "\n",
    "tp, fp, fn, tn, acc, prec, rec = evaluate(probabilities, y_test, threshold=0.5)\n",
    "\n",
    "print('Confusion matrix at threshold 0.5')\n",
    "print(f'  True positives  (cancers caught):   {tp}')\n",
    "print(f'  False negatives (cancers missed):   {fn}   <-- the dangerous mistake')\n",
    "print(f'  False positives (false alarms):     {fp}')\n",
    "print(f'  True negatives  (correctly cleared):{tn}')\n",
    "print()\n",
    "print(f'Accuracy:  {acc:.1%}   (baseline was {baseline_accuracy:.1%})')\n",
    "print(f'Precision: {prec:.1%}')\n",
    "print(f'Recall:    {rec:.1%}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbb6cdb1",
   "metadata": {},
   "source": [
    "## 10) Which metric should we prioritize?\n",
    "\n",
    "The two kinds of mistake are **not equally bad**:\n",
    "\n",
    "- A **false negative** means a patient with cancer is sent home. The disease progresses untreated.\n",
    "- A **false positive** means a healthy patient gets an extra examination. Inconvenient and stressful, but recoverable.\n",
    "\n",
    "So in this task we prioritize **recall of the malignant class**: catching as many real cancers as possible. We are willing to accept a few more false alarms, and even a slightly **lower accuracy**, to gain true positives. A model that is 1% less accurate but misses fewer cancers is the better medical tool.\n",
    "\n",
    "**The decision threshold is the tuning knob for this trade.** The default 0.5 is not a law of nature. If we lower the threshold (for example: flag every tumor with $p \\ge 0.3$), the model flags more tumors: recall goes up (more cancers caught), while precision tends to go down, and accuracy can fall too (more false alarms). Raising the threshold does the opposite. The next cell shows the whole trade in one table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25809df5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2026-11-03T20:23:27.618315Z",
     "start_time": "2026-11-03T20:23:27.613215Z"
    }
   },
   "outputs": [],
   "source": [
    "print(' threshold |  TP |  FN |  FP |  TN | recall | precision | accuracy')\n",
    "print('-' * 68)\n",
    "for threshold in [0.9, 0.7, 0.5, 0.3, 0.1]:\n",
    "    tp, fp, fn, tn, acc, prec, rec = evaluate(probabilities, y_test, threshold)\n",
    "    print(f'   {threshold:4.1f}    | {tp:3d} | {fn:3d} | {fp:3d} | {tn:3d} | {rec:6.1%} |  {prec:6.1%}  |  {acc:6.1%}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5b41828",
   "metadata": {},
   "source": [
    "Read the table from top to bottom: as the threshold falls, false negatives (missed cancers) shrink and recall rises, while false positives grow. Accuracy is highest somewhere in the middle, but the **highest-accuracy threshold is not automatically the best medical choice**.\n",
    "\n",
    "One careful reading: a recall of 100% in this table means all 42 cancers **in this particular test set**. It is not a guarantee about future patients, especially from a sample this small.\n",
    "\n",
    "In a real project, this threshold is chosen together with the medical experts, based on the actual costs of each mistake, and it is tuned on a **validation set**, never on the final test set (the rule from Week 2)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce9e2329",
   "metadata": {},
   "source": [
    "## 11) Inference: one new patient\n",
    "\n",
    "Inference means using the trained model on a patient it has never seen. We do it in two small steps. First we pick **one patient** from the test set and look at their measurements, so the input is something concrete, not an abstract row number. Then we ask the model for its prediction.\n",
    "\n",
    "One sentence to remember for real-world use: the model's output is a **probability, not a diagnosis**. A system like this supports a doctor's decision; it does not replace it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79762975",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2026-11-03T20:23:45.432425Z",
     "start_time": "2026-11-03T20:23:45.422092Z"
    }
   },
   "outputs": [],
   "source": [
    "patient_number = 1          # patient number 1 in the test set (you can try 0 to 113)\n",
    "\n",
    "# Show this patient's measurements as a small table (the first 8 of the 30 columns).\n",
    "patient_table = pd.DataFrame({\n",
    "    'measurement': X_df.columns,\n",
    "    'value': X_test[patient_number],\n",
    "})\n",
    "patient_table.head(8)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cbec976",
   "metadata": {},
   "source": [
    "These are the patient's real, unscaled measurements. Now we ask the model. Remember the two rules from the earlier labs: the model expects the **scaled** version of the measurements, and it expects them as a table with one row. Both are already prepared in `X_test_t`, so we only need to take the right row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff51df3a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2026-11-03T20:24:11.589067Z",
     "start_time": "2026-11-03T20:24:11.583263Z"
    }
   },
   "outputs": [],
   "source": [
    "# Take patient 1 as a one-row table (that is what patient_number : patient_number + 1 does).\n",
    "one_patient = X_test_t[patient_number : patient_number + 1]\n",
    "\n",
    "# Ask the model. Its output is the probability that the tumor is malignant.\n",
    "with torch.no_grad():\n",
    "    probability = model(one_patient).item()\n",
    "\n",
    "print('Predicted probability of malignant:', round(probability, 3))\n",
    "\n",
    "if probability >= 0.5:\n",
    "    print('Decision at threshold 0.5:  FLAG for examination')\n",
    "else:\n",
    "    print('Decision at threshold 0.5:  no flag')\n",
    "\n",
    "if y_test[patient_number] == 1:\n",
    "    print('True diagnosis:             malignant')\n",
    "else:\n",
    "    print('True diagnosis:             benign')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8425871d",
   "metadata": {},
   "source": [
    "The model gives this patient a very high probability of malignant, the decision rule flags the tumor, and the true diagnosis confirms it. Now change `patient_number` to **0** in the first cell of this section and run both cells again: that patient is benign, and you will see the probability drop to almost zero and the flag disappear."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9c07354",
   "metadata": {},
   "source": [
    "## 12) What you learned\n",
    "\n",
    "- Binary classification changes three things compared with regression: a **probability output** (logit plus sigmoid), the **binary cross-entropy loss**, and the **evaluation**.\n",
    "- Cross-entropy punishes confident wrong answers heavily, which trains honest probabilities.\n",
    "- The confusion matrix gives the four outcomes; in this task the false negative (a missed cancer) is the costly mistake, so **recall of the malignant class is the metric to prioritize**, even at the price of some accuracy.\n",
    "- The decision threshold is a tuning knob: lowering it trades false alarms for caught cancers.\n",
    "\n",
    "### Exercises\n",
    "\n",
    "1. Find the highest threshold in the table cell that reaches **100% recall** on the test set. What precision and accuracy come with it? Would you accept this trade in a hospital? (Note: for convenience we are choosing the threshold on the test set, which breaks the rule from section 10; in a real project this choice is made on a separate validation set.)\n",
    "2. Remove both `nn.ReLU()` lines and retrain. Compare with the same experiment in Lab 2. If the result surprises you, think about what it says about this dataset rather than about activation functions.\n",
    "3. Replace the threshold 0.5 in the inference cell with your answer from exercise 1, then check patient 0 and patient 1 again. Does the decision change for either of them?"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "env_pytorch",
   "language": "python",
   "name": "env_pytorch"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}