{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial 2 - Perceptron and logistic regression\n",
    "\n",
    "In this tutorial, we'll introduce logistic regression and the percepron learning algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "from sklearn.datasets import make_classification\n",
    "from sklearn.linear_model import LogisticRegression, Perceptron\n",
    "from sklearn.metrics import accuracy_score\n",
    "import sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Binary classification\n",
    "\n",
    "This section will introduce the specific context of binary classification, which is a sub-category of supervised learning. While the previous tutorial focused more on regression, where the model predicts a real-valued output $y \\in \\mathbb{R}$, this tutorial focuses on classification, where the model assigns the input into one of several classes. For binary classification, we formalize this as predicting one class from a finite set of two elements, typically $y \\in \\{0, 1\\}$\n",
    "\n",
    "### Generating synthetic data for classification\n",
    "\n",
    "We will generate synthetic data points suitable for a two-class problem, ensuring the input vectors $\\vec{x} \\in \\mathbb{R}^d$ and corresponding labels $y \\in \\{0, 1\\}$ are created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = make_classification(\n",
    "    n_samples = 100,\n",
    "    n_features = 2,\n",
    "    n_redundant = 0,\n",
    "    n_informative = 2,\n",
    "    n_classes = 2,\n",
    "    random_state = 42\n",
    ")\n",
    "\n",
    "print(\"Input shape X:\", X.shape)\n",
    "print(\"Output shape y:\", y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize the generated data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(8, 6))\n",
    "plt.scatter(X[:, 0], X[:, 1], c = y, marker = 'o', label = 'Data Points')\n",
    "plt.title('Synthetic Data for Binary Classification')\n",
    "plt.xlabel('Feature 1')\n",
    "plt.ylabel('Feature 2')\n",
    "plt.grid(True)\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A logistic regression model\n",
    "\n",
    "Logistic regression is an improvement over linear regression. Whereas linear regression could only fit linear functions for regression, logistic regression augments it with a nonlinearity making it suitable for classification.\n",
    "\n",
    "In linear regression, the relationship is defined by a simple dot product: $ f ( \\vec{x} ) = \\vec{x}^T \\vec{w} + b$. Logistic regression uses a similar linear combination but passes the result through a sigmoid (logistic) function:\n",
    "\n",
    "$$ \\sigma ( x ) = \\frac{1}{1 + e^{-x}} $$\n",
    "\n",
    "This function is used to map the output to a probability between 0 and 1, suitable for classification.\n",
    "\n",
    "### Cost function\n",
    "\n",
    "In contrast to linear regression, which typically uses the Mean Square Error (MSE) cost function, classification models use cost functions better suited for probability outputs. For binary classification, the standard cost function is the Binary Cross-Entropy (or Log Loss)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def binary_cross_entropy(y_true, y_pred_proba):\n",
    "    # Ensure clipping to avoid log(0)\n",
    "    epsilon = 1e-12\n",
    "    y_pred_proba = np.clip(y_pred_proba, epsilon, 1. - epsilon)\n",
    "    # Log Loss calculation\n",
    "    return -np.mean(y_true * np.log(y_pred_proba) + (1 - y_true) * np.log(1 - y_pred_proba))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementation and visualization of a decision strategy\n",
    "\n",
    "The output of logistic regression is a probability $\\hat{p}$. The decision strategy is a threshold applied to this probability to assign a final class label (0 or 1). Typically, if $\\hat{p} \\geq 0.5$, the prediction is class 1; otherwise, it is class 0. The boundary where $\\hat{p} = 0.5$ is called the decision boundary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_decision_boundary(model, X, y):    \n",
    "    # Setup the limits and create a meshgrid\n",
    "    x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5\n",
    "    y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5\n",
    "\n",
    "    # Create coordinate matrices for the plot grid\n",
    "    xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.02),\n",
    "                         np.arange(y_min, y_max, 0.02))\n",
    "\n",
    "    # Predict class labels for every point on the meshgrid    \n",
    "    Z = model.predict(np.c_[xx.ravel(), yy.ravel()])\n",
    "    Z = Z.reshape(xx.shape)\n",
    "\n",
    "    plt.figure(figsize=(8, 6))\n",
    "    plt.contourf(xx, yy, Z, alpha=0.4, cmap=plt.cm.coolwarm)\n",
    "    \n",
    "    # Plot the training points\n",
    "    plt.scatter(X[:, 0], X[:, 1], c=y, edgecolors='k', marker='o', cmap=plt.cm.coolwarm, s=50)\n",
    "    plt.xlabel('Feature 1')\n",
    "    plt.ylabel('Feature 2')\n",
    "    plt.xlim(xx.min(), xx.max())\n",
    "    plt.ylim(yy.min(), yy.max())\n",
    "    plt.grid(True)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training logistic regression with scikit-learn\n",
    "\n",
    "We will use the `scikit-learn` library to train the logistic regression model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = LogisticRegression() \n",
    "model.fit(X, y)\n",
    "y_pred_optimal_proba = model.predict_proba(X)[:, 1] # Get probabilities for class 1\n",
    "optimal_cost = binary_cross_entropy(y, y_pred_optimal_proba)\n",
    "\n",
    "print(\"Optimal cost (BCE) after training:\", optimal_cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also visualize the learned decision boundary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_decision_boundary(model, X, y) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating the Model\n",
    "\n",
    "For classification, evaluation focuses on metrics like accuracy, precision, and recall, rather than just MSE or $R^2$ which are typically used for regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred_classes = model.predict(X)\n",
    "acc = accuracy_score(y, y_pred_classes)\n",
    "print(f\"Model Accuracy: {acc:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 1: Manual logistic regression\n",
    "\n",
    "Train a logistic regression model using gradient descent on BCE cost function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ManualLogisticRegression:\n",
    "    def __init__(self, learning_rate=0.01, epochs=1000):\n",
    "        self.learning_rate = learning_rate\n",
    "        self.epochs = epochs\n",
    "\n",
    "        self.W = None\n",
    "        self.b = None\n",
    "\n",
    "    def _sigmoid(self, z):\n",
    "        z = np.clip(z, -500, 500)\n",
    "        return 1 / (1 + np.exp(-z))\n",
    "\n",
    "    def fit(self, X_train, y_train):\n",
    "        n_samples, n_features = X_train.shape\n",
    "        \n",
    "        # Initialize parameters\n",
    "        self.W = np.zeros(n_features) \n",
    "        self.b = 0\n",
    "        \n",
    "        # Gradient Descent loop\n",
    "        for epoch in range(self.epochs):\n",
    "            # TODO\n",
    "        \n",
    "        return self\n",
    "\n",
    "    def predict_proba(self, X_input):\n",
    "        linear_output = np.dot(X_input, self.W) + self.b\n",
    "        return self._sigmoid(linear_output)\n",
    "\n",
    "    def predict(self, X_input):\n",
    "        probabilities = self.predict_proba(X_input)\n",
    "        return (probabilities >= 0.5).astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = ManualLogisticRegression() \n",
    "model.fit(X, y)\n",
    "y_pred_optimal_proba = model.predict_proba(X)\n",
    "optimal_cost = binary_cross_entropy(y, y_pred_optimal_proba)\n",
    "\n",
    "print(\"Optimal cost (BCE) after training:\", optimal_cost)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_decision_boundary(model, X, y) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's again also look at the accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred_classes = model.predict(X)\n",
    "acc = accuracy_score(y, y_pred_classes)\n",
    "print(f\"Model Accuracy: {acc:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The perceptron model\n",
    "\n",
    "The perceptron is an earlier, foundational algorithm for linear binary classification. In essence, it's a simple algorithm that mimics a single neuron in the human brain, capable of making basic decisions and, crucially, learning from its mistakes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Perceptron()\n",
    "model.fit(X, y)\n",
    "plot_decision_boundary(model, X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Perceptron learning algorithm and implementation\n",
    "\n",
    "The perceptron operates by adjusting weights based on misclassified points. It does this through a process called the *Perceptron Learning Algorithm*, which is a form of supervised learning. This means we provide the perceptron with training data that includes both the inputs and the correct outputs. The perceptron then adjusts its weights and bias to make its predictions more accurate.\n",
    "- Initialization: The perceptron starts with random values for its weights and bias.\n",
    "- Prediction: It takes an example from the training data and makes a prediction based on its current weights and bias.\n",
    "- Error Calculation: The perceptron's prediction is compared to the correct output from the training data. The difference between the two is the error.\n",
    "- Weight and Bias Update: This is the crucial learning step.\n",
    "    - If the prediction is correct: The weights and bias are not changed.\n",
    "    - If the prediction is incorrect: The weights and bias are adjusted to reduce the error. The update rule is simple: the change in a weight is proportional to the error and the input it's associated with. A small value called the \"learning rate\" controls how much the weights and bias are adjusted in each step.\n",
    "\n",
    "$$ w_{i, t} = w_{i, t-1} + \\eta ( y_i - \\hat{y_i} ) \\vec{x}_i $$\n",
    "\n",
    "- Iteration: This process of predicting, calculating the error, and updating the weights is repeated for all the examples in the training data, multiple times. With each pass through the data (called an \"epoch\"), the perceptron's weights and bias get closer to the optimal values that allow it to make correct predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 2\n",
    "\n",
    "Create a manual perceptron model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ManualPerceptron:\n",
    "    def __init__(self, learning_rate=0.01, epochs=100):\n",
    "        self.learning_rate = learning_rate\n",
    "        self.epochs = epochs\n",
    "\n",
    "        self.weights = None\n",
    "        self.bias = None\n",
    "\n",
    "    def fit(self, X, y):\n",
    "        n_samples, n_features = X.shape\n",
    "        \n",
    "        # Initialize parameters\n",
    "        self.weights = np.zeros(n_features)\n",
    "        self.bias = 0\n",
    "\n",
    "        for epoch in range(self.epochs):\n",
    "            for x_i, y_i in zip(X, y):\n",
    "                # TODO\n",
    "\n",
    "    def predict(self, X_input):\n",
    "        linear_output = np.dot(X_input, self.weights) + self.bias\n",
    "        y_pred = np.where(linear_output >= 0, 1, 0)\n",
    "        \n",
    "        return y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = ManualPerceptron()\n",
    "model.fit(X, y)\n",
    "plot_decision_boundary(model, X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating the Model\n",
    "\n",
    "For classification, evaluation focuses on metrics like accuracy, precision, and recall, rather than just MSE or $R^2$ which are typically used for regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred_classes = model.predict(X)\n",
    "acc = accuracy_score(y, y_pred_classes)\n",
    "print(f\"Model Accuracy: {acc:.4f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "nsn",
   "language": "python",
   "name": "nsn"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}