{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial 1 - Neural network intro\n",
    "\n",
    "In this tutorial, we'll introduce the **Python** programming language and some of its libraries, most notably [PyTorch](https://pytorch.org/).\n",
    "\n",
    "The approaches used by PyTorch are broadly applicable to other frameworks as well, such as [TensorFlow](https://www.tensorflow.org/), [JAX](https://docs.jax.dev/en/latest/), [Flux.jl](https://fluxml.ai/) or [Deep Learning Toolbox](https://www.mathworks.com/products/deep-learning.html).\n",
    "\n",
    "For a more detailed overview of PyTorch you can also go through the [oficial tutorials](https://pytorch.org/tutorials/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Package installation with uv and Jupyter\n",
    "\n",
    "We'll first install the [uv](https://docs.astral.sh/uv/) package manager. We can install Jupyter system-wide with uv by running:\n",
    "\n",
    "```bash\n",
    "$ uv tool install jupyterlab\n",
    "```\n",
    "\n",
    "We need to create a uv project and connect it to Jupyter:\n",
    "```bash\n",
    "$ uv init\n",
    "$ uv add --dev ipykernel\n",
    "$ uv run ipython kernel install --user --env VIRTUAL_ENV $(pwd)/.venv --name=tzn\n",
    "```\n",
    "\n",
    "In order for the newly installed Jupyter kernel to be available, we need to restart Jupyter.\n",
    "\n",
    "Finally, we can install the required packages:\n",
    "```bash\n",
    "$ uv add datasets gensim matplotlib numpy scikit-learn torch torch-cluster torch-geometric torchvision transformers\n",
    "```\n",
    "\n",
    "Note: `torch-cluster` must be compiled from source against the installed version of PyTorch. uv handles this automatically, but requires the following section in `pyproject.toml`:\n",
    "```toml\n",
    "[tool.uv.extra-build-dependencies]\n",
    "torch-cluster = [\"torch\"]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 1\n",
    "\n",
    "Run the aforementioned steps to create a package and have the next cell run without errors. It should print something like `Torch version: 2.8.0`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch.utils.data import DataLoader\n",
    "from torchvision import datasets, transforms\n",
    "\n",
    "print(\"Torch version:\", torch.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensors and working with them in PyTorch\n",
    "\n",
    "Let's first take a look at how to create and manipulate tensors and what is happening in the background in order to automatically compute gradients. \n",
    "\n",
    "A tensor is in essence a name for an array with $n$ dimentisons - i. e. like the `ndarray` in numpy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create a PyTorch tensor by converting from python or numpy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = torch.tensor([[1, 2], [3,4]])\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also convert it back to numpy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(a.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A tensor always has the attributes `shape` and `dtype`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Shape:', a.shape)\n",
    "print('Element data type:', a.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other ways of constructing a tensor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(torch.ones((1, 3)))\n",
    "print(torch.zeros((1, 3)))\n",
    "print(torch.randn((1, 4))) # Normal distribution\n",
    "print(torch.rand((1, 4))) # Uniform distribution on [0, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can do basic operations with PyTorch tensors. Let's construct three tensors with the same dtype:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = torch.tensor([[1, 2], [3,4]], dtype = torch.float32)\n",
    "b = torch.ones((2, 1))\n",
    "c = torch.randn((2, 1))\n",
    "print(a)\n",
    "print(b)\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Matrix multiplication:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(a.matmul(b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Element-wise addition:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(b.add(c))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Element-wise square:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(a.square())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Euclidean norm of a vector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(b.norm())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Matrix rank:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(torch.linalg.matrix_rank(a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Element-wise application of the sine function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(torch.sin(a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Boolean indicator of whether each element is greater than 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(a.greater(1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 2 - basics of working with data\n",
    "\n",
    "Let's load the [MNIST](https://en.wikipedia.org/wiki/MNIST_database) dataset and calculate the average value of each feature. Running the next cell for the first time downloads the dataset and may take a while."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = datasets.MNIST(\n",
    "    root = \"data\",\n",
    "    download = True,\n",
    "    transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize(0, 1)])\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've loaded images from the MNIST dataset. Now try to calculate the average value of each feature accross all images. Use the [`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader) class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automatic gradient calculation\n",
    "\n",
    "To automatically calculate gradients, we use the regular `torch.Tensor`. To calculate gradients with respect to some variable, we need to set it to `requires_grad = True`. This will make PyTorch track all operations which depend on this variable. To actually calculate the gradient, we need to call the `fun.backward()` method, which will calculate th derivative of `fun` with respect to all the input variables. If `fun` is of greater dimension than 1, we need to pass to `backward` a tensor with repspect to which to calculate the Jacobian."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor([1, 2, 3], dtype = torch.float32, requires_grad = True)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = torch.square(x)\n",
    "f.backward(torch.ones_like(f))\n",
    "\n",
    "print('Derivative of x^2:', x.grad)\n",
    "with torch.no_grad():\n",
    "    print('2*x:', 2*x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor([1, 2, 3], dtype = torch.float32, requires_grad = True)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = torch.sin(x)\n",
    "f.backward(torch.ones_like(f))\n",
    "    \n",
    "print('Derivative of sin(x):', x.grad)\n",
    "with torch.no_grad():\n",
    "    print('cos(x):', torch.cos(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic neural net on the MNIST dataset\n",
    "\n",
    "Let's define the basic parameters of the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 50\n",
    "hidden_layer_width = 100\n",
    "output_width = 10\n",
    "learning_rate = 0.01"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load the training and testing data and create a `DataLoader` for each:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_train = datasets.MNIST(\n",
    "    root = \"data\",\n",
    "    train = True,\n",
    "    download = True,\n",
    "    transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize(0, 1)])\n",
    ")\n",
    "data_test = datasets.MNIST(\n",
    "    root = \"data\",\n",
    "    train = False,\n",
    "    download = True,\n",
    "    transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize(0, 1)])\n",
    ")\n",
    "\n",
    "dataloader_train = DataLoader(data_train, batch_size = batch_size, shuffle = True)\n",
    "dataloader_test = DataLoader(data_test, batch_size = batch_size, shuffle = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll create a simple neural network using `torch.nn.Sequential`, which is a class to put several layers into sequentially in the order of their insertion.\n",
    "\n",
    "The input layer will be a `torch.nn.Flatten`, which will convert the 2D images into vectors.\n",
    "\n",
    "`torch.nn.Sequential` is a specialization of the more general [`tf.nn.Module`](https://pytorch.org/docs/main/generated/torch.nn.Module.html) class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = nn.Sequential()\n",
    "model.append(nn.Flatten())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's add one hidden layer with the `tanh` activation function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.append(nn.Linear(data_train.data.shape[1] * data_train.data.shape[2], hidden_layer_width))\n",
    "model.append(nn.Tanh())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's add an output layer with 10 neurons (= number of classes) and the softmax activation function: \n",
    "$$ \\sigma \\left( \\vec{z} \\right)_i = \\frac{e^{\\vec{z}_i}}{\\sum_{j = 1}^K e^{\\vec{z}_j}} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.append(nn.Linear(hidden_layer_width, output_width))\n",
    "model.append(nn.Softmax(dim=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Printing a model will also give us a summary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Such a model may also be directly applied to data liek a function.\n",
    "\n",
    "Its outputs will be probability vectors. Because we haven't trained the model yet, the results will be random."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model(data_train[0][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final prediction of the model could be realized e.g. as the element with the highest probability:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10, 7))\n",
    "for i in range(24):\n",
    "    plt.subplot(4, 6, i + 1)\n",
    "    plt.xticks([])\n",
    "    plt.yticks([])\n",
    "    plt.grid(False)\n",
    "    plt.imshow(data_train[i][0].reshape(28, 28), cmap = plt.cm.bone)\n",
    "    plt.title(model(data_train[i][0]).detach().argmax(axis = 1).item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the model\n",
    "\n",
    "Now we can actually train the model. We'll need two things for that:\n",
    "* The *loss* function - the function that we want to minimize\n",
    "* The *optimizer* - a function that uses the gradient to make one optimization step\n",
    "\n",
    "Note:\n",
    "* `torch.nn.NLLLoss` is the cross-entropy loss function which takes as input a probability distribution\n",
    "* `torch.nn.CrossEntropyLoss` is the cross-netropy loss function, which only takes as inputs class weights (i. e. they don't need to sum up to 1)\n",
    "\n",
    "In other words, `CrossEntropyLoss` combines `NLLLoss` and `Softmax`.\n",
    "\n",
    "We will use the ADAM optimizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_fn = nn.NLLLoss()\n",
    "optimizer = torch.optim.Adam(model.parameters())\n",
    "epochs = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train the model we will use the `DataLoader`. We will train in mini-batches, which means that we will split the datset into smaller batches, calculate the average loss function for the whole batch and then use it to make one optimization step. One epoch then means using the whole training dataset once (which includes many optimization steps).\n",
    "\n",
    "Let us define 3 functions needed to train the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_accuracy(model, dataloader):\n",
    "    num_correct = 0\n",
    "    \n",
    "    with torch.no_grad():\n",
    "        for (X, y) in dataloader:\n",
    "            pred = model(X)\n",
    "            num_correct += (pred.argmax(1) == y).type(torch.float).sum().item()\n",
    "\n",
    "    accuracy = num_correct / len(dataloader.dataset)\n",
    "    return accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_loss_accuracy(model, dataloader, loss_fn):\n",
    "    loss = 0\n",
    "    num_correct = 0\n",
    "    \n",
    "    with torch.no_grad():\n",
    "        for (X, y) in dataloader:\n",
    "            pred = model(X)\n",
    "            loss += loss_fn(pred, y).item()\n",
    "            num_correct += (pred.argmax(1) == y).type(torch.float).sum().item()\n",
    "\n",
    "    loss /= len(dataloader)\n",
    "    accuracy = num_correct / len(dataloader.dataset)\n",
    "    return loss, accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_model(model, loss_fn, optimizer, epochs, dataloader_train, dataloader_test, early_stopper = None, log_period = 10000):\n",
    "    for epoch in range(epochs):\n",
    "        processed_since_log = 0\n",
    "        for batch, (X, y) in enumerate(dataloader_train):\n",
    "            model.train()\n",
    "            pred = model(X)\n",
    "            loss = loss_fn(pred, y)\n",
    "\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "            optimizer.zero_grad()\n",
    "\n",
    "            processed_since_log += dataloader_train.batch_size\n",
    "\n",
    "            if processed_since_log >= log_period:\n",
    "                current = min((batch + 1) * dataloader_train.batch_size, len(data_train))\n",
    "                loss = loss.item()\n",
    "                model.eval()\n",
    "                train_acc = calculate_accuracy(model, dataloader_train)\n",
    "                test_loss, test_acc = calculate_loss_accuracy(model, dataloader_test, loss_fn)\n",
    "                print(f\"train loss: {loss:>7f}  test loss: {test_loss:>7f}  train accuracy: {train_acc:>3f}  test accuracy: {test_acc:>3f}  [sample {current:>5d}/{len(data_train):>5d}] [epoch {epoch+1:>2d}/{epochs:>2d}]\")\n",
    "                processed_since_log -= log_period"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_model(model, loss_fn, optimizer, epochs, dataloader_train, dataloader_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating the model\n",
    "\n",
    "To evaluate the model on the test set, we can reuse the previously defined functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.eval()\n",
    "test_loss, test_acc = calculate_loss_accuracy(model, dataloader_test, loss_fn)\n",
    "print('Test loss:', test_loss)\n",
    "print('Test accuracy:', test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we need the \"raw\" output values, we may again just use the model as a function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = model(data_test[0][0])\n",
    "print('Prediction shape:', predictions.shape)\n",
    "print('Probablity of the first image being a zero:', predictions[0, 0].item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To obtain actual class predictions, we will use the `torch.argmax` function to find out which digit has the highest probability:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_pred = predictions.detach().argmax(dim = 1)\n",
    "print('Probabilities for the first image:', predictions.detach())\n",
    "print('Most likely label for the first image:', Y_pred[0].item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10, 7))\n",
    "for i in range(24):\n",
    "    plt.subplot(4, 6, i + 1)\n",
    "    plt.xticks([])\n",
    "    plt.yticks([])\n",
    "    plt.grid(False)\n",
    "    plt.imshow(data_test[i][0].reshape(28, 28), cmap = plt.cm.bone)\n",
    "    plt.title(model(data_test[i][0]).detach().argmax(axis = 1).item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 3 - Creating the model manually\n",
    "\n",
    "Let's again define the basic properties of the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 50\n",
    "hidden_layer_width = 100\n",
    "output_width = 10\n",
    "learning_rate = 0.01\n",
    "epochs = 10\n",
    "loss_fn = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define the parameters we will need for a neural network with one hidden layer with 100 neurons:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "W1 = (0.1 * torch.randn((data_train[0][0].shape[1] * data_train[0][0].shape[2], hidden_layer_width))).clone().requires_grad_(True)\n",
    "b1 = torch.zeros((hidden_layer_width,), requires_grad = True)\n",
    "W2 = (0.1 * torch.randn((hidden_layer_width, output_width))).clone().requires_grad_(True)\n",
    "b2 = torch.zeros((output_width,), requires_grad = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define the forward pass of the neural net in a function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(inputs):\n",
    "    # TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's train the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "log_period = 10000\n",
    "\n",
    "for epoch in range(epochs):\n",
    "    processed_since_log = 0\n",
    "    for batch, (X, y) in enumerate(dataloader_train):\n",
    "        pred = predict(X)\n",
    "        loss = loss_fn(pred, y)\n",
    "        loss.backward()\n",
    "\n",
    "        for variable in [W1, b1, W2, b2]:\n",
    "            with torch.no_grad():\n",
    "                variable -= variable.grad * learning_rate\n",
    "            variable.grad = None\n",
    "\n",
    "        processed_since_log += dataloader_train.batch_size\n",
    "\n",
    "        if processed_since_log >= log_period:\n",
    "            current = min((batch + 1) * dataloader_train.batch_size, len(data_train))\n",
    "            loss = loss.item()\n",
    "            train_acc = calculate_accuracy(predict, dataloader_train)\n",
    "            test_loss, test_acc = calculate_loss_accuracy(predict, dataloader_test, loss_fn)\n",
    "            print(f\"train loss: {loss:>7f}  test loss: {test_loss:>7f}  train accuracy: {train_acc:>3f}  test accuracy: {test_acc:>3f}  [sample {current:>5d}/{len(data_train):>5d}] [epoch {epoch+1:>2d}/{epochs:>2d}]\")\n",
    "            processed_since_log -= log_period"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training (hyper-)parameters\n",
    "\n",
    "In the previous examples we just set the various hyper-parameters of the network semi-randomly. Let's now take a look at them and their possible values.\n",
    "\n",
    "### Network architecture\n",
    "\n",
    "- Number of layers\n",
    "- Layer widths\n",
    "- activation functions\n",
    "    - linear\n",
    "    - tanh\n",
    "    - sigmoid\n",
    "    - hard sigmoid\n",
    "    - relu\n",
    "    - selu\n",
    "    - softmax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20, 20))\n",
    "i = 1\n",
    "for activationFunction in [nn.Tanh(), nn.Hardtanh(), nn.Sigmoid(), nn.Hardsigmoid(), nn.ReLU(), nn.LeakyReLU(), nn.SELU(), nn.ELU()]:\n",
    "    plt.subplot(4, 4, i)\n",
    "    i += 1\n",
    "    plt.grid(True)\n",
    "    xs = torch.linspace(-5, 5, 100);\n",
    "    ys = activationFunction(xs)\n",
    "    plt.plot(xs, ys)\n",
    "    plt.title(type(activationFunction).__name__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loss functions\n",
    "\n",
    "- Mean square error\n",
    "    $$ \\operatorname{MSE}=\\frac{1}{n}\\sum_{i=1}^n(Y_i-\\hat{Y_i})^2. $$\n",
    "- Hinge\n",
    "    $$ \\operatorname{Hinge} = \\sum_{i=1}^K \\max(0, 1- Y_i \\cdot \\hat{Y_i}) $$\n",
    "- Cross-entropy\n",
    "    $$ \\operatorname{crossentropy} = - \\sum_{i=1}^K Y_i \\cdot \\log(\\hat{Y_i}) $$\n",
    "\n",
    "## Optimization algorithms\n",
    "\n",
    "- SGD\n",
    "- RMSProp\n",
    "- Adagrad\n",
    "- Adam\n",
    "- Adadelta\n",
    "- Adamax\n",
    "- Nadam\n",
    "- AdamW\n",
    "\n",
    "### Learning schedule\n",
    "\n",
    "We may optionally add a so-called weight decay to the optimization algorithm. This means that the learning rate will become smaller as we train. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "initial_learning_rate = 0.01\n",
    "batch_count = 64\n",
    "optimizer = torch.optim.SGD(model.parameters(), weight_decay=0.001)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More advanced algorithms such as AdamW have weight decay built-in.\n",
    "\n",
    "## Using the training dataset\n",
    "\n",
    "We can set 2 ways in which the training dataset is used:\n",
    "- Epochs: How many times we loop over the whole dataset\n",
    "- Batching: Using the dataset not sample-by-sample but in batches\n",
    "\n",
    "# Task 4\n",
    "\n",
    "Try to modify the previously defined model on MNIST to obtain the best possible results by tuning its hyper-parameters."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tzn",
   "language": "python",
   "name": "tzn"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}