Welcome to the Philips deep learning course! Have you ever wondered how computers can "see" and understand images? This course is designed to introduce you to the fascinating world of deep learning, with a specific focus on its powerful applications in computer vision. We will build our skills from the ground up, equipping you to ask insightful questions and contribute effectively when discussing new product ideas involving deep learning.

Lesson 1: Linear and Logistic Regression¶

In our first lesson, we'll start with foundational concepts by exploring linear and logistic regression. You might ask, why begin with such seemingly simple models when our goal is deep learning?

There are several compelling reasons:

• Building Blocks: Neural networks can be understood as powerful compositions of these basic models. Linear and logistic regression introduce fundamental concepts crucial for building neural networks, such as loss functions (measuring how well our model performs), weight matrices (the parameters our models learn), output functions (transforming the model's output), and optimization (finding the best parameters).
• Illustrating Core Ideas: These simpler models provide clear and intuitive examples to demonstrate more abstract but critical concepts like overfitting (when a model performs too well on training data but poorly on new data), underfitting (when a model is too simple to capture the data's patterns), and the bias-variance tradeoff (the inherent conflict between a model's simplicity and its ability to fit the data). Understanding these concepts with linear and logistic regression makes it easier to grasp them when we move to more complex neural network architectures.

Important notes¶

1. It is fine to consult with colleagues to solve the problems, in fact it is encouraged.
2. Please turn off AI tools, we want you to memorize concepts and not just quickly breeze through problems. To turn off AI click on the gear in the top right corner. got to AI assistance -> Untick Show AI powered inline completions, Untick consented to use generative AI features, tick Hide Generative AI features

1. Linear Regression¶

Linear regression is a statistical method used to model the relationship between a dependent variable and one or more independent variables by fitting a linear equation to observed data. It is important to have at least seen linear regression once before diving into neural networks. This is because it familiarizes you with important concepts: Weights, Bias Terms (not to be confused with the more colloquial use of the term bias), Target Values, and some basic linear algebra operations that you will need later.

Goal of Linear Regression:

The primary goal of linear regression is to find the "best-fit" line (or hyperplane in higher dimensions) that minimizes the difference between the observed values and the values predicted by the linear model. This difference is typically measured using the Mean Squared Error (MSE). By minimizing the MSE, we aim to find the parameters of the linear equation that best represent the underlying relationship in the data.

Matrices in Linear Regression:

To represent the data and the model parameters, we typically use matrices:

• y (Target values): This is a column vector containing the dependent variable values (the thing we want to predict). It has dimensions $n \times 1$, where $n$ is the number of observations.

$$ \mathbf{y} = \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix} $$

• X (Data matrix): This is a matrix containing the independent variable values (the stuff from which we are predicting). To include a bias term in the model, we add a column of ones to the left of the independent variable data. It has dimensions $n \times (d+1)$, where $n$ is the number of observations and $d$ is the number of independent variables.

$$ \mathbf{X} = \begin{bmatrix} 1 & x_{1,1} & x_{1,2} & \dots & x_{1,d} \\ 1 & x_{2,1} & x_{2,2} & \dots & x_{2,d} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & x_{n,1} & x_{n,2} & \dots & x_{n,d} \end{bmatrix} $$

• w (Coefficients/Weights): This is a column vector containing the coefficients (including the bias term) that we want to learn from the data. It has dimensions $(d+1) \times 1$. This is the part of the regression that will be changed during our fitting of the line.

$$ \mathbf{w} = \begin{bmatrix} w_0 \\ w_1 \\ \vdots \\ w_d \end{bmatrix} $$

Finding the solution for Mean the Squared Error:

The linear regression model can be expressed as:

$$ \hat{\mathbf{y}} = \mathbf{X} \mathbf{w} $$

where $\hat{\mathbf{y}}$ are the predicted values. Basically, you multiply your weight matrix $\mathbf{w}$ with your data matrix $\mathbf{X}$, to get the predictions $\hat{\mathbf{y}}$.

The Mean Squared Error (MSE) is defined as:

$$ MSE = \frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2 $$

It measures the difference between your predictions and the real target values. Each difference is then squared to make sure it is positive. In matrix form, the sum of squared errors (SSE) is:

$$ SSE = (\mathbf{y} - \mathbf{X} \mathbf{w})^T (\mathbf{y} - \mathbf{X} \mathbf{w}) $$

To find the analytical solution for the weights $\mathbf{w}$ that minimize the SSE (and thus the MSE), we take the derivative of the SSE with respect to $\mathbf{w}$, set it to zero, and solve for $\mathbf{w}$:

$$ \frac{\partial SSE}{\partial \mathbf{w}} = \frac{\partial}{\partial \mathbf{w}} (\mathbf{y} - \mathbf{X} \mathbf{w})^T (\mathbf{y} - \mathbf{X} \mathbf{w}) = \mathbf{0} $$

Expanding and differentiating, we get:

$$ -2 \mathbf{X}^T (\mathbf{y} - \mathbf{X} \mathbf{w}) = \mathbf{0} $$$$ -2 \mathbf{X}^T \mathbf{y} + 2 \mathbf{X}^T \mathbf{X} \mathbf{w} = \mathbf{0} $$$$ \mathbf{X}^T \mathbf{X} \mathbf{w} = \mathbf{X}^T \mathbf{y} $$

Assuming $(\mathbf{X}^T \mathbf{X})$ is invertible, the analytical solution for the weights $\mathbf{w}$ is:

$$ \mathbf{w} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{y} $$

You can see that the Moore-Penrose Pseudoinverse emerges in this equation.

Interesting Fact: This equation is known as the Normal Equation, and it provides a closed-form solution for the optimal weights in linear regression.

Exercise 1.1: Implementing linear regression¶

As our first exercise, we are going to implement finding the optimal weight matrix ($\mathbf{w}$). In order to do this we will first have to generate some toy data $\mathbf{X}$ and $\mathbf{y}$.

2. Logistic Regression¶

The Problem¶

While linear regression is used for predicting continuous values (like a house price), logistic regression (specifically binary logistic regression) is used for classification problems. The goal is to predict which of two distinct categories or classes (typically labeled 0 and 1, like "spam" or "not spam," or "malignant" or "benign") a given data point belongs to.

Unlike linear regression, where we found a direct formula to calculate the best weights (this direct formula is what we call an analytical solution), there is no such simple formula for finding the optimal weights in logistic regression. This is primarily because of the sigmoid function (or logistic function) which we use at the end of our model. Logistic regression primarily familiarizes us with output/activation functions (in this case the sigmoid), optimization (in this case Stochastic Gradient Descent), and predicting probabilities.

The logistic regression model works by first calculating a linear combination of the inputs and weights. A linear combination is simply multiplying each input feature by its corresponding weight and summing them up, just like we did in the linear regression equation ($z = w_0 \cdot 1 + w_1 \cdot x_1 + w_2 \cdot x_2 + \dots$). We can represent this in matrix form as:

$$ z = \mathbf{X} \mathbf{w} $$

After calculating this linear combination ($z$), we apply the sigmoid function to it. The sigmoid function is a mathematical function that takes any real-valued number and squashes it into a value between 0 and 1. The equation for the sigmoid function is:

$$ \sigma(z) = \frac{1}{1 + e^{-z}} $$

So, the predicted probability $\hat{y}$ that the data point belongs to class 1 is given by:

$$ \hat{y} = \sigma(z) = \frac{1}{1 + e^{-z}} $$

The output of the sigmoid function ($\hat{y}$) can be interpreted as the probability of the data point belonging to class 1. To make a final class prediction, we typically use a threshold (a cutoff value, often 0.5). If the calculated probability ($\hat{y}$) is greater than or equal to this threshold, we classify the data point as class 1; otherwise, we classify it as class 0.

When we start talking about neural networks later, $\mathbf{z}$ are often referred to as logits and $\hat{y}$ are referred to as your probabilities.

So what does this look like?

As you can see, the sigmoid function squishes the values of Y to be within the 0 to 1 range.

The Binary Cross Entropy Loss¶

In linear regression we used a loss function called "mean square error". However, now we are working with probabilities instead of normal values.

There are several reasons for not using the mean squared error, most are beyond the scope of this course. However, a very important practical reason is that you want to make sure that overconfidence is penalized. If our logistic regression model makes a very confident but incorrect prediction, the mean squared error does not penalize this behaviour.

Instead, for classification problems like logistic regression, we use a different measure called "cross-entropy", which does penalize the model for overconfident predictions.

For binary classification (where there are only two classes, like 0 or 1), the formula for cross-entropy loss for a single data point is:

$$ \text{Loss} = -(y \log(\hat{y}) + (1 - y) \log(1 - \hat{y})) $$

Here we have:

1. $y$ -> The actual class or "ground truth". In our case it is 0 or 1
2. $\hat{y}$ -> The predicted class probability (between 0 and 1).

If the actual class $y$ is 1, the formula becomes $-\log(\hat{y})$, which is small if $\hat{y}$ is close to 1 and large if $\hat{y}$ is close to 0. If the actual class $y$ is 0, the formula becomes $-\log(1 - \hat{y})$, which is small if $\hat{y}$ is close to 0 and large if $\hat{y}$ is close to 1.

(Stochastic) Gradient Descent¶

In logistic regression, because we use that curved "S"-shaped function (the sigmoid) at the end, we don't have such a simple, analytical solution to find the best weights. Hence, we can't just calculate the answer in one step.

So, instead of a direct calculation, we have to use a different approach. Imagine you're trying to find the lowest point in a valley while blindfolded. You can't just walk directly there. What you'd do is start somewhere, feel around a bit to see which direction goes downhill, take a small step in that direction, and repeat the process. You keep taking small steps downhill until you can't go down any further.

That's similar to what we do in logistic regression to find the best weights. We start with some initial guess for the weights. Then, we repeatedly adjust these weights in small steps, trying to make our model better at predicting the classes and reducing the "loss" (which, as we discussed, is measured by cross-entropy). This process of starting with a guess and gradually improving it over many steps is called an iterative optimization method. We keep iterating, or repeating the steps, until we find the set of weights that minimizes our error.

Now, let's go back to our valley analogy. How do you know which direction is downhill? You'd feel the slope or the steepness of the ground. In machine learning, the equivalent of feeling the slope is calculating something called the gradient.

The gradient tells us the direction in which the loss function is increasing most steeply. Since we want to find the lowest point (where the loss is minimized), we need to move in the opposite direction of the steepest increase. Think of it as following the path that goes straight downhill. This process of moving in the direction opposite to the gradient to decrease the loss is the core idea behind gradient descent.

So, with gradient descent, we start with our initial guess for the weights, calculate the gradient of the loss function at that point, and then adjust the weights by taking a step in the opposite direction of the gradient.

How big of a step should we take? That's controlled by a crucial parameter called the learning rate (in deep learning it is typically denoted as $\lambda$). The learning rate is a small positive number that determines how much we update the weights based on the gradient.

• A small learning rate means we take tiny steps. This makes the process slow, like cautiously tiptoeing downhill. However, it increases the chances of finding the absolute lowest point without overshooting it.
• A large learning rate means we take big steps, like leaping down the hillside. This can make the process faster, but there's a risk of overshooting the lowest point or even bouncing back and forth around it without ever settling down.

The goal is to choose a learning rate that is just right – allowing us to reach the minimum loss efficiently without being too aggressive. We repeat this process of calculating the gradient and updating the weights using the learning rate over many iterations until we reach a point where the loss is as low as possible.

let's look at two ways to perform gradient descent: standard Gradient Descent (GD) and Stochastic Gradient Descent (SGD).

In standard Gradient Descent (GD), to figure out which direction is downhill (to calculate the gradient), we use all of our data points. We calculate the error for every single data point, sum up how these errors would change if we slightly adjusted the weights, and then take a step based on that total sum. While this gives us a very accurate picture of the true "downhill" direction based on all the data, it can be very computationally expensive if we have a really large dataset. Imagine having millions or billions of data points – calculating the gradient using all of them for every single step would take a very long time!

This is where Stochastic Gradient Descent (SGD) comes in. The word "stochastic" basically means "involving a random variable." In SGD, instead of using the entire dataset to calculate the gradient, we use just one single data point (or sometimes a very small group of data points, called a "mini-batch") at each step. We calculate the gradient based on this one (or few) data points and immediately update the weights. This reduces the computational cost, but the steps are noisier. As a result of this noise our path downward will be more zigzaggy than with regular GD.

Update Equations for Stochastic Gradient Descent¶

In SGD, we update the weights iteratively based on the gradient calculated from a single data point (or a small batch, in which it is called batch gradient descent). The formula for updating the weights at each step is:

$$ w_{new} = w_{old} - \lambda \times \nabla L(y_i, \hat{y}_i, w) $$

Let's break down what each part of this formula means:

• $w_{new}$: These are the new, updated weights that we calculate in the current step. These will be used in the next iteration.
• $w_{old}$: These are the current weights (the weights from the previous iteration) before we update them. We start with some initial weights and update them repeatedly using this formula.
• $\lambda$ is our learning rate. The small positive number that controls how big of a step we take in the direction of the gradient.
• $\nabla L(y_i, \hat{y}_i, w)$: This is the gradient of the loss function for a single data point $(x_i, y_i)$ with respect to the weights $w$. The loss function $L$ measures how wrong our prediction $\hat{y}_i$ is compared to the actual value $y_i$ for that specific data point, given the current weights $w$. The gradient $\nabla L$ tells us the direction of the steepest increase in the loss for this single data point. By subtracting the learning rate times the gradient, we move the weights in the direction that decreases the loss for that data point.

The one thing we are missing is the definition of the gradient $\nabla L(y_i, \hat{y}_i, w)$. We will give a brief derivation of it below. If you are interested in the full derivation, scroll down to the appendix.

Remember, we have our binary cross entropy that we seek to minimize at each time step $i$:

$$ \text{Loss} = -(y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i))$$

We want to minimize it by making the predicted values $\hat{y}$ very close to the true values $y$, by changing the weight matrix $\mathbf{W}$. If you remember from high school, we can take the derivative to get a step in the right direction. But we cannot do this directly, because there is also a sigmoid function in the way. So we have to use the chain rule. Another way of writing $\nabla L(y_i, \hat{y}_i, w)$ is $ \frac{\delta L}{\delta w}$.

$$\frac{\delta L}{\delta w} = \frac{\delta L}{\delta \hat{y}_i}\frac{\delta \hat{y}_i}{\delta z_i}\frac{\delta z_i}{\delta w} \quad\quad (1)$$

That means we have to take three derivatives:

1. The derivative of the binary cross entropy loss w.r.t the probabilities:

$$\frac{\delta L}{\delta \hat{y}_i} = \frac{\hat{y}_i - y_i}{\hat{y}_i (1 - \hat{y}_i)} $$

2. The derivative of the sigmoid with respect to the logits $z$:

$$ \frac{\delta \hat{y}_i}{\delta z_i} =\hat{y_i}(1 - \hat{y}_i)$$

3. The derivative of $z$ with respect to the weights $w$:

$$ \frac{\partial z_i}{\partial \mathbf{w}} = \mathbf{x_i}^T $$

When we put all of these together using formula $(1)$. We get the following expression:

$$ \frac{\delta L}{\delta w} = \frac{\hat{y}_i - y_i}{\hat{y}_i (1 - \hat{y}_i)}(\hat{y_i}(1 - \hat{y}_i)) \mathbf{x_i}^T \quad \quad (2) $$

This simplifies to:

$$ \frac{\delta L}{\delta w} = (\hat{y}_i - y_i)\mathbf{x_i}^T \quad \quad (3) $$

Plug that back into our update equation, and we get:

$$ w_{new} = w_{old} - \lambda (\hat{y}_i - y_i)\mathbf{x_i}^T$$

This lies at heart of our optimization process. We sample a datapoint $i$, feed multiply it with the weight matrix to get our prediction $\hat{y}$ and update it with the weights. Do this often enough and you should see the model becoming more and more accurate.

Exercise 1.2 Exploring the Iris dataset¶

We are going to work with a "real" dataset about flowers, how nice! This dataset contains data about three different species of flowers, their sepal width, length and petal width and lenght. The first thing you are going to do is answer a few questions about this dataset after loading it.

Questions

1. How many datapoints are there in this dataset?
2. Excluding the bias, how many weights will we have in our weight matrix (i.e. what will the dimension be)?

Now that you have answered the questions, we are going to start implementing logistic regression. But before we do that we have to remove one of the species from our iris dataset. This is because there are three species of flower, and our current form of logistic regression can only handle 2 classes at the same time. So let's start with that.

Good, now that we have a cleaned dataset (with a bias term appended to it), we can start setting up our model. If you do it correctly, you should see the cross entropy loss decreasing.

Exercise 1.3: Learning Rate experiment¶

Now that we have a working model, we are going to run a few experiments. I would like to see you try different values for the learning rate and report back on the accuracy (defined below, just call the function with the arguments). I have also provided a list below with different values for the learning rates that you can try. Report back on the accuracies and answer the questions once you are done.

1. What do you notice about the relationship between the learning rate and the accuracy?

2. Bonus: If you run the experiment more than one time, you may find different results. Do you have any idea why this is?

3. Bonus: Sometimes the experiments result in 100% accuracy, do you think that if we get data that the model has not seen yet it will also get 100% accuracy, why (not)?

3. Softmax Regression¶

In the last section, we've played around with logistic regression and the binary cross entropy loss. These are specific cases of something called softmax regression and the cross entropy loss. The latter are used in the case there are more than two classes.

The softmax output function is given (for a datapoint i and a class c out of a total of K classes) given by:

$$ \sigma_{softmax} (z_{i, c}) = \frac{e^{z_{i, c}}}{\sum^K_{i, j = 1} e^{z_{i, j}}}$$

The (multi-class) cross entropy loss is given (for a class c and a single datapoint i) by:

$$ \mathcal{L}_{CE} = - \sum^K_{c = 1} y_c \log(\hat{y}_c)$$

Where $y$ is the target label, and $\hat{y}$ is the predicted label. One thing in softmax regression that is different from logistic regression is that instead of having a vector with weights (and a bias). You now have a $K\times M$ matrix, where $K$ are the number of classes and $M$ are the the number of weights + 1. Intuitively, what happens during softmax is that you calculate the probability of each class, then you take the maximum of that and that is your prediction. This exercise teaches you how to stack weight vectors on top of each other and use those instead of single regression vectors, which is something that we do in deep learning all the time.

Example of prediction in softmax regression¶

Let's illustrate a forward pass (this is, making predictions) in softmax regression with 3 classes using matrix operations, similar to what we would do with the Iris dataset (which has 4 features + a bias term).

Assume we have a data matrix $\mathbf{X}$ with $n$ data points and $d=4$ features, plus a column of ones for the bias term, so its dimensions are $n \times 5$.

$$ \mathbf{X} = \begin{bmatrix} 1 & x_{1,1} & x_{1,2} & x_{1,3} & x_{1,4} \\ 1 & x_{2,1} & x_{2,2} & x_{2,3} & x_{2,4} \\ \vdots & \vdots & \vdots & \vdots & \vdots \\ 1 & x_{n,1} & x_{n,2} & x_{n,3} & x_{n,4} \end{bmatrix} $$

We have a weight matrix $\mathbf{W}$ with dimensions $(d+1) \times K$, where $d+1=5$ (features + bias) and $K=3$ (number of classes, i.e. the different kinds of flowers in our dataset). Each column of $\mathbf{W}$ corresponds to the weight vector for a specific class.

$$ \mathbf{W} = \begin{bmatrix} w_{0,1} & w_{0,2} & w_{0,3} \\ w_{1,1} & w_{1,2} & w_{1,3} \\ w_{2,1} & w_{2,2} & w_{2,3} \\ w_{3,1} & w_{3,2} & w_{3,3} \\ w_{4,1} & w_{4,2} & w_{4,3} \end{bmatrix} $$

The first step of the forward pass is to calculate the linear combinations (logits) for each data point and each class by performing matrix multiplication:

$$ \mathbf{Z} = \mathbf{X} \mathbf{W} $$

The dimensions of $\mathbf{Z}$ will be $n \times K$ (i.e., $n \times 3$). Each element $z_{i,c}$ in $\mathbf{Z}$ is the linear combination for data point $i$ and class $c$.

$$ \mathbf{Z} = \begin{bmatrix} z_{1,1} & z_{1,2} & z_{1,3} \\ z_{2,1} & z_{2,2} & z_{2,3} \\ \vdots & \vdots & \vdots \\ z_{n,1} & z_{n,2} & z_{n,3} \end{bmatrix} $$

The second step is to apply the softmax function to each row of the logits matrix $\mathbf{Z}$. The softmax function is applied independently to each data point's vector of logits. For each data point $i$, the predicted probability for class $c$, $\hat{y}_{i,c}$, is calculated as:

$$ \hat{y}_{i,c} = \frac{e^{z_{i,c}}}{\sum_{j=1}^{K} e^{z_{i,j}}} $$

Applying the softmax function to every element of $\mathbf{Z}$ results in the predicted probability matrix $\hat{\mathbf{Y}}$, which has the same dimensions as $\mathbf{Z}$ ($n \times 3$). Each element $\hat{y}_{i,c}$ represents the predicted probability that data point $i$ belongs to class $c$. The values in each row of $\hat{\mathbf{Y}}$ will sum up to 1.

$$ \hat{\mathbf{Y}} = \sigma_{softmax} (\mathbf{Z}) = \begin{bmatrix} \hat{y}_{1,1} & \hat{y}_{1,2} & \hat{y}_{1,3} \\ \hat{y}_{2,1} & \hat{y}_{2,2} & \hat{y}_{2,3} \\ \vdots & \vdots & \vdots \\ \hat{y}_{n,1} & \hat{y}_{n,2} & \hat{y}_{n,3} \end{bmatrix} $$

The matrix $\hat{\mathbf{Y}}$ contains the predicted probability distribution over the three classes for each of the $n$ data points.

Softmax Update Equations¶

In order to jump straight into implementation, here are the equations you will need to implement softmax regression. First, the gradient:

$$\nabla \mathcal{L}_{CE}(\mathbf{x}_i, \mathbf{y}_i, \mathbf{W}) =\mathbf{x}_i(\mathbf{\hat{y}}_{i} - \mathbf{y}_{i})^T $$

Because the gradient is taken with respect to each class, we would like to make sure that updates are easy to do. This is the vectorized respresentation of the gradient update for a randomly sampled datapoint $i$:

In our Iris dataset example, the vector $\mathbf{x}_i$ would be a $5 \times 1$ vector, $(\mathbf{\hat{y}}_{i} - \mathbf{y}_{i})^T$ would be a $1\times 3$ vector. If the outer product is taken between these two vectors, you should get a $5\times3$ matrix. This is the same size as our weight matrix. The final update equation becomes:

$$\mathbf{w}_{new} = \mathbf{w}_{old} - \lambda \mathbf{x}_i(\mathbf{\hat{y}}_{i} - \mathbf{y}_{i})^T $$

If you are interested in the derivations behind the softmax update, please check the appendix.

Exercise 1.4 Implement softmax regression¶

You are going to implement (parts of softmax regression). In the cells below I have indicated what you should implement yourself. Look at the equations above to see what you have to do!

Hint: Be mindful of how numpy handles outer products, the transposes in the gradient calculation are slightly different.

If your loss function decreases to about 0.1, you are good!

End of Tutorial¶

Now that you have completed this tutorial, you have been introduced to many important topics: Weight Matrices, Loss functions (Means Squared Error and Binary Cross Entropy), optimization techniques (Stochastic Gradient Descent), learning rates, and output functions (The sigmoid activation function and the softmax activation function). These will set you up nicely to learn more about deep learning.

Appendix: Derivation of binary logistic regression updates.¶

Derivative of the sigmoid function¶

The sigmoid function is defined as:

$$ \sigma(z) = \frac{1}{1 + e^{-z}} $$

To find the derivative of the sigmoid function with respect to $z$, which is $\frac{\partial \sigma(z)}{\partial z}$ or $\frac{\partial \hat{y}}{\partial z}$, we can use the quotient rule or rewrite the expression. Let's use the latter approach:

$$ \sigma(z) = (1 + e^{-z})^{-1} $$

Now, applying the chain rule for differentiation:

$$ \frac{\partial \sigma(z)}{\partial z} = -1 \cdot (1 + e^{-z})^{-2} \cdot \frac{\partial}{\partial z}(1 + e^{-z}) $$$$ \frac{\partial \sigma(z)}{\partial z} = - (1 + e^{-z})^{-2} \cdot (-e^{-z}) $$$$ \frac{\partial \sigma(z)}{\partial z} = \frac{e^{-z}}{(1 + e^{-z})^2} $$

We can rewrite this expression in terms of $\sigma(z)$:

$$ \frac{e^{-z}}{(1 + e^{-z})^2} = \frac{1 + e^{-z} - 1}{(1 + e^{-z})^2} = \frac{1 + e^{-z}}{(1 + e^{-z})^2} - \frac{1}{(1 + e^{-z})^2} $$$$ = \frac{1}{1 + e^{-z}} - \left(\frac{1}{1 + e^{-z}}\right)^2 $$$$ = \sigma(z) - (\sigma(z))^2 $$$$ = \sigma(z)(1 - \sigma(z)) $$

So, the derivative of the sigmoid function with respect to $z$ is:

$$ \frac{\partial \hat{y}}{\partial z} = \sigma(z)(1 - \sigma(z)) = \hat{y}(1 - \hat{y}) $$

Derivative of the binary cross entropy¶

The Binary Cross-Entropy loss for a single data point $(x_i, y_i)$ is given by:

$$ L(y_i, \hat{y}_i) = -(y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i)) $$

To find the derivative of the loss with respect to the predicted probability $\hat{y}_i$, which is $\frac{\partial L}{\partial \hat{y}_i}$, we differentiate the loss function with respect to $\hat{y}_i$:

$$ \frac{\partial L}{\partial \hat{y}_i} = - \frac{\partial}{\partial \hat{y}_i} (y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i)) $$

Using the fact that $\frac{\partial}{\partial x} \log(x) = \frac{1}{x}$:

$$ \frac{\partial L}{\partial \hat{y}_i} = - \left( y_i \frac{\partial}{\partial \hat{y}_i} \log(\hat{y}_i) + (1 - y_i) \frac{\partial}{\partial \hat{y}_i} \log(1 - \hat{y}_i) \right) $$

For the second term:

$$ \frac{\partial}{\partial \hat{y}_i} \log(1 - \hat{y}_i) = \frac{1}{1 - \hat{y}_i} \cdot (-1) = - \frac{1}{1 - \hat{y}_i} $$

Substituting these back into the derivative of the loss:

$$ \frac{\partial L}{\partial \hat{y}_i} = - \left( y_i \frac{1}{\hat{y}_i} + (1 - y_i) \left( - \frac{1}{1 - \hat{y}_i} \right) \right) $$$$ \frac{\partial L}{\partial \hat{y}_i} = - \left( \frac{y_i}{\hat{y}_i} - \frac{1 - y_i}{1 - \hat{y}_i} \right) $$

To simplify, we find a common denominator:

$$ \frac{\partial L}{\partial \hat{y}_i} = - \left( \frac{y_i (1 - \hat{y}_i) - \hat{y}_i (1 - y_i)}{\hat{y}_i (1 - \hat{y}_i)} \right) $$$$ \frac{\partial L}{\partial \hat{y}_i} = - \left( \frac{y_i - y_i \hat{y}_i - \hat{y}_i + y_i \hat{y}_i}{\hat{y}_i (1 - \hat{y}_i)} \right) $$$$ \frac{\partial L}{\partial \hat{y}_i} = - \left( \frac{y_i - \hat{y}_i}{\hat{y}_i (1 - \hat{y}_i)} \right) $$

Finally, distributing the negative sign:

$$ \frac{\partial L}{\partial \hat{y}_i} = \frac{\hat{y}_i - y_i}{\hat{y}_i (1 - \hat{y}_i)} $$

So, the derivative of the Binary Cross-Entropy loss with respect to the predicted probability is:

$$ \frac{\partial L}{\partial \hat{y}_i} = \frac{\hat{y}_i - y_i}{\hat{y}_i (1 - \hat{y}_i)} $$

Derivative of the softmax function¶

In softmax regression, instead of outputting a single probability like in binary logistic regression, we output a probability distribution over multiple classes. The softmax function takes a vector of real numbers (the linear combinations, or "logits", for each class) and converts them into a probability distribution, where the probabilities for all classes sum up to 1.

For a data point $i$ and a class $c$, the output of the softmax function, denoted as $\hat{y}_{i,c}$, is calculated as follows:

$$ \hat{y}_{i,c} = \frac{e^{z_{i,c}}}{\sum_{j=1}^{K} e^{z_{i,j}}} $$

Where:

• $z_{i,c}$ is the linear combination of inputs and weights for data point $i$ and class $c$. This is calculated as $z_{i,c} = \mathbf{x}_i \mathbf{w}_c$, where $\mathbf{x}_i$ is the input vector for data point $i$ and $\mathbf{w}_c$ is the weight vector for class $c$.
• $K$ is the total number of classes.
• The denominator is the sum of the exponentiated linear combinations for all classes for data point $i$. This ensures that the output probabilities for all classes sum up to 1.

We will consider two cases:

Case 1: When $j = c$ (The target class is the same as the predicted class)

We need to find $\frac{\partial \hat{y}_{i,c}}{\partial z_{i,c}}$. Using the quotient rule $\left(\frac{u}{v}\right)' = \frac{u'v - uv'}{v^2}$, where $u = e^{z_{i,c}}$ and $v = \sum_{k=1}^{K} e^{z_{i,k}}$:

$\frac{\partial u}{\partial z_{i,c}} = e^{z_{i,c}}$

$\frac{\partial v}{\partial z_{i,c}} = \frac{\partial}{\partial z_{i,c}} \left( e^{z_{i,1}} + e^{z_{i,2}} + \dots + e^{z_{i,c}} + \dots + e^{z_{i,K}} \right) = e^{z_{i,c}}$ (since only the term with $z_{i,c}$ is dependent on $z_{i,c}$)

Now, applying the quotient rule:

$$ \frac{\partial \hat{y}_{i,c}}{\partial z_{i,c}} = \frac{(e^{z_{i,c}})(\sum_{k=1}^{K} e^{z_{i,k}}) - (e^{z_{i,c}})(e^{z_{i,c}})}{(\sum_{k=1}^{K} e^{z_{i,k}})^2} $$

We can factor out $e^{z_{i,c}}$ from the numerator:

$$ \frac{\partial \hat{y}_{i,c}}{\partial z_{i,c}} = \frac{e^{z_{i,c}} \left( \sum_{k=1}^{K} e^{z_{i,k}} - e^{z_{i,c}} \right)}{(\sum_{k=1}^{K} e^{z_{i,k}})^2} $$

Now, let's rewrite this in terms of $\hat{y}_{i,c}$:

$$ \frac{\partial \hat{y}_{i,c}}{\partial z_{i,c}} = \frac{e^{z_{i,c}}}{\sum_{k=1}^{K} e^{z_{i,k}}} \cdot \frac{\sum_{k=1}^{K} e^{z_{i,k}} - e^{z_{i,c}}}{\sum_{k=1}^{K} e^{z_{i,k}}} = \hat{y}_{i,c} \left( 1 - \frac{e^{z_{i,c}}}{\sum_{k=1}^{K} e^{z_{i,k}}} \right) $$$$ \frac{\partial \hat{y}_{i,c}}{\partial z_{i,c}} = \hat{y}_{i,c} (1 - \hat{y}_{i,c}) $$

This result is very similar to the derivative of the sigmoid function!

Case 2: When $j \neq c$ (The output class is different from the linear combination class)

We need to find $\frac{\partial \hat{y}_{i,j}}{\partial z_{i,c}}$ where $j \neq c$. Using the quotient rule again, with $u = e^{z_{i,j}}$ and $v = \sum_{k=1}^{K} e^{z_{i,k}}$:

$\frac{\partial u}{\partial z_{i,c}} = 0$ (since $j \neq c$, $e^{z_{i,j}}$ does not depend on $z_{i,c}$)

$\frac{\partial v}{\partial z_{i,c}} = e^{z_{i,c}}$

Applying the quotient rule:

$$ \frac{\partial \hat{y}_{i,j}}{\partial z_{i,c}} = \frac{(0)(\sum_{k=1}^{K} e^{z_{i,k}}) - (e^{z_{i,j}})(e^{z_{i,c}})}{(\sum_{k=1}^{K} e^{z_{i,k}})^2} $$$$ \frac{\partial \hat{y}_{i,j}}{\partial z_{i,c}} = - \frac{e^{z_{i,j}} e^{z_{i,c}}}{(\sum_{k=1}^{K} e^{z_{i,k}})^2} $$

We can rewrite this in terms of softmax outputs:

$$ \frac{\partial \hat{y}_{i,j}}{\partial z_{i,c}} = - \frac{e^{z_{i,j}}}{\sum_{k=1}^{K} e^{z_{i,k}}} \cdot \frac{e^{z_{i,c}}}{\sum_{k=1}^{K} e^{z_{i,k}}} = - \hat{y}_{i,j} \hat{y}_{i,c} $$

So, the derivative of the softmax output for class $j$ with respect to the linear combination for class $c$ (when $j \neq c$) is:

$$ \frac{\partial \hat{y}_{i,j}}{\partial z_{i,c}} = - \hat{y}_{i,j} \hat{y}_{i,c} $$

In summary, the derivative of the softmax output $\hat{y}_{i,j}$ with respect to the linear combination $z_{i,c}$ is:

$$ \frac{\partial \hat{y}_{i,j}}{\partial z_{i,c}} = \begin{cases} \hat{y}_{i,c}(1 - \hat{y}_{i,c}) & \text{if } j = c \\ -\hat{y}_{i,j}\hat{y}_{i,c} & \text{if } j \neq c \end{cases} $$

Derivative of the Cross entropy function¶

For multi-class classification problems, we use the cross-entropy loss function. For a single data point $i$ and its true class label $y_i$ (represented as a one-hot encoded vector), and the predicted probability distribution $\hat{\mathbf{y}}_i$ (output of the softmax function), the cross-entropy loss is defined as:

$$ \mathcal{L}_{CE} = - \sum_{c=1}^{K} y_{i,c} \log(\hat{y}_{i,c}) $$

Where:

• $\mathcal{L}_{CE}$ is the cross-entropy loss for data point $i$.
• $K$ is the total number of classes.
• $y_{i,c}$ is the $c$-th element of the one-hot encoded true label vector for data point $i$. This will be 1 for the true class and 0 for all other classes.
• $\hat{y}_{i,c}$ is the predicted probability that data point $i$ belongs to class $c$ (output of the softmax function for class $c$).

Since $y_{i,c}$ is 1 only for the true class (let's say the true class is $t$), the sum simplifies to just the term for the true class:

$$ \mathcal{L}_{CE} = - \log(\hat{y}_{i,t}) $$

where $\hat{y}_{i,t}$ is the predicted probability for the true class $t$. The goal of minimizing this loss function is to maximize the predicted probability of the true class.

We want to find the partial derivative of $\mathcal{L}_{CE}$ with respect to the predicted probability for a specific class $c$, which is $\frac{\partial \mathcal{L}_{CE}}{\partial \hat{y}_{i,c}}$.

$$ \frac{\partial \mathcal{L}_{CE}}{\partial \hat{y}_{i,c}} = \frac{\partial}{\partial \hat{y}_{i,c}} \left( - \sum_{j=1}^{K} y_{i,j} \log(\hat{y}_{i,j}) \right) $$

We can move the negative sign outside the derivative and the derivative inside the sum:

$$ \frac{\partial \mathcal{L}_{CE}}{\partial \hat{y}_{i,c}} = - \sum_{j=1}^{K} \frac{\partial}{\partial \hat{y}_{i,c}} (y_{i,j} \log(\hat{y}_{i,j})) $$

Now, we differentiate each term in the sum with respect to $\hat{y}_{i,c}$. Remember that $y_{i,j}$ is a constant with respect to $\hat{y}_{i,c}$.

The derivative $\frac{\partial}{\partial \hat{y}_{i,c}} (y_{i,j} \log(\hat{y}_{i,j}))$ is non-zero only when $j = c$. In this case, the derivative is:

$$ \frac{\partial}{\partial \hat{y}_{i,c}} (y_{i,c} \log(\hat{y}_{i,c})) = y_{i,c} \frac{\partial}{\partial \hat{y}_{i,c}} (\log(\hat{y}_{i,c})) = y_{i,c} \cdot \frac{1}{\hat{y}_{i,c}} $$

For all other cases where $j \neq c$, the term $y_{i,j} \log(\hat{y}_{i,j})$ does not contain $\hat{y}_{i,c}$, so its derivative with respect to $\hat{y}_{i,c}$ is 0.

Therefore, the sum simplifies to just the term where $j=c$:

$$ \frac{\partial \mathcal{L}_{CE}}{\partial \hat{y}_{i,c}} = - \left( y_{i,c} \cdot \frac{1}{\hat{y}_{i,c}} \right) $$

So, the partial derivative of the cross-entropy loss with respect to the predicted probability for class $c$ is:

$$ \frac{\partial \mathcal{L}_{CE}}{\partial \hat{y}_{i,c}} = - \frac{y_{i,c}}{\hat{y}_{i,c}} $$

This derivative tells us how the cross-entropy loss changes as the predicted probability for a specific class $c$ changes.

Derivation of the Gradient of the softmax function with Cross entropy Loss¶

We are using the chain rule to find the gradient of the loss $\mathcal{L}_{CE}$ with respect to the weight vector for class $c$, $\mathbf{w}_c$:

$$ \frac{\partial \mathcal{L}_{CE}}{\partial \mathbf{w}_c} = \sum_{j=1}^{K} \frac{\partial \mathcal{L}_{CE}}{\partial \hat{y}_{i,j}} \times \frac{\partial \hat{y}_{i,j}}{\partial z_{i,c}} \times \frac{\partial z_{i,c}}{\partial \mathbf{w}_c} $$

We have the following individual derivatives:

1. $\frac{\partial \mathcal{L}_{CE}}{\partial \hat{y}_{i,j}} = - \frac{y_{i,j}}{\hat{y}_{i,j}}$ (Derivative of Cross-Entropy Loss w.r.t. Softmax Output)

2. $\frac{\partial \hat{y}_{i,j}}{\partial z_{i,c}} = \begin{cases} \hat{y}_{i,c}(1 - \hat{y}_{i,c}) & \text{if } j = c \\ -\hat{y}_{i,j}\hat{y}_{i,c} & \text{if } j \neq c \end{cases}$ (Derivative of Softmax Output w.r.t. Linear Combination)

3. $\frac{\partial z_{i,c}}{\partial \mathbf{w}_c} = \mathbf{x}_i^T$ (Derivative of Linear Combination w.r.t. Weights for class $c$)

Now, let's substitute these into the chain rule formula. The sum over $j$ needs to consider the two cases for $\frac{\partial \hat{y}_{i,j}}{\partial z_{i,c}}$:

$$ \frac{\partial \mathcal{L}_{CE}}{\partial \mathbf{w}_c} = \left( \frac{\partial \mathcal{L}_{CE}}{\partial \hat{y}_{i,c}} \times \frac{\partial \hat{y}_{i,c}}{\partial z_{i,c}} \times \frac{\partial z_{i,c}}{\partial \mathbf{w}_c} \right) + \sum_{j \neq c}^{K} \left( \frac{\partial \mathcal{L}_{CE}}{\partial \hat{y}_{i,j}} \times \frac{\partial \hat{y}_{i,j}}{\partial z_{i,c}} \times \frac{\partial z_{i,c}}{\partial \mathbf{w}_c} \right) $$

Let's substitute the derivatives for each part:

For the $j = c$ term: $$ \left( - \frac{y_{i,c}}{\hat{y}_{i,c}} \right) \times \left( \hat{y}_{i,c}(1 - \hat{y}_{i,c}) \right) \times \left( \mathbf{x}_i^T \right) = - y_{i,c}(1 - \hat{y}_{i,c}) \mathbf{x}_i^T $$

For the $j \neq c$ terms: $$ \sum_{j \neq c}^{K} \left( - \frac{y_{i,j}}{\hat{y}_{i,j}} \right) \times \left( - \hat{y}_{i,j}\hat{y}_{i,c} \right) \times \left( \mathbf{x}_i^T \right) = \sum_{j \neq c}^{K} y_{i,j} \hat{y}_{i,c} \mathbf{x}_i^T $$

Now, sum the terms:

$$ \frac{\partial \mathcal{L}_{CE}}{\partial \mathbf{w}_c} = - y_{i,c}(1 - \hat{y}_{i,c}) \mathbf{x}_i^T + \sum_{j \neq c}^{K} y_{i,j} \hat{y}_{i,c} \mathbf{x}_i^T $$

Factor out $\mathbf{x}_i^T$:

$$ \frac{\partial \mathcal{L}_{CE}}{\partial \mathbf{w}_c} = \left( - y_{i,c}(1 - \hat{y}_{i,c}) + \sum_{j \neq c}^{K} y_{i,j} \hat{y}_{i,c} \right) \mathbf{x}_i^T $$$$ \frac{\partial \mathcal{L}_{CE}}{\partial \mathbf{w}_c} = \left( - y_{i,c} + y_{i,c}\hat{y}_{i,c} + \sum_{j \neq c}^{K} y_{i,j} \hat{y}_{i,c} \right) \mathbf{x}_i^T $$

Since $y_{i,j}$ is 0 for all classes except the true class (let's say the true class is $t$), the sum $\sum_{j \neq c}^{K} y_{i,j}$ is 0 if $c$ is the true class, and 1 if $c$ is not the true class (because only the $y_{i,t}$ term will be 1).Let's simplify the approach.

Let's go back to the form:

$$ \frac{\partial \mathcal{L}_{CE}}{\partial z_{i,c}} = \sum_{j=1}^{K} \frac{\partial \mathcal{L}_{CE}}{\partial \hat{y}_{i,j}} \frac{\partial \hat{y}_{i,j}}{\partial z_{i,c}} $$

Substitute the derivatives:

$$ \frac{\partial \mathcal{L}_{CE}}{\partial z_{i,c}} = \sum_{j=1}^{K} \left( - \frac{y_{i,j}}{\hat{y}_{i,j}} \right) \left (\hat{y}_{i,j} (\delta_{jc} - \hat{y}_{i,c}) \right) $$

Where $\delta_{jc}$ is the Kronecker delta, which is 1 if $j=c$ and 0 otherwise.

$$ \frac{\partial \mathcal{L}_{CE}}{\partial z_{i,c}} = - \sum_{j=1}^{K} y_{i,j} (\delta_{jc} - \hat{y}_{i,c}) $$$$ \frac{\partial \mathcal{L}_{CE}}{\partial z_{i,c}} = - \sum_{j=1}^{K} (y_{i,j}\delta_{jc} - y_{i,j}\hat{y}_{i,c}) $$$$ \frac{\partial \mathcal{L}_{CE}}{\partial z_{i,c}} = - \left( \sum_{j=1}^{K} y_{i,j}\delta_{jc} - \sum_{j=1}^{K} y_{i,j}\hat{y}_{i,c} \right) $$

The term $\sum_{j=1}^{K} y_{i,j}\delta_{jc}$ is equal to $y_{i,c}$ because $\delta_{jc}$ is only non-zero when $j=c$.

The term $\sum_{j=1}^{K} y_{i,j}\hat{y}_{i,c} = \hat{y}_{i,c} \sum_{j=1}^{K} y_{i,j}$. Since $y_{i,j}$ is a one-hot encoded vector for the true class, $\sum_{j=1}^{K} y_{i,j} = 1$. So, $\sum_{j=1}^{K} y_{i,j}\hat{y}_{i,c} = \hat{y}_{i,c}$.

Substituting these back:

$$ \frac{\partial \mathcal{L}_{CE}}{\partial z_{i,c}} = - (y_{i,c} - \hat{y}_{i,c}) = \hat{y}_{i,c} - y_{i,c} $$

Now, to get the gradient with respect to the weights $\mathbf{w}_c$, we multiply by $\frac{\partial z_{i,c}}{\partial \mathbf{w}_c}$:

$$ \frac{\partial \mathcal{L}_{CE}}{\partial \mathbf{w}_c} = \frac{\partial \mathcal{L}_{CE}}{\partial z_{i,c}} \times \frac{\partial z_{i,c}}{\partial \mathbf{w}_c} = (\hat{y}_{i,c} - y_{i,c}) \mathbf{x}_i^T $$

This is the gradient of the cross-entropy loss with respect to the weight vector for class $c$.

So, the gradient for the weights of class $c$ is:

$$ \nabla_{\mathbf{w}_c} \mathcal{L}_{CE} = (\hat{y}_{i,c} - y_{i,c}) \mathbf{x}_i^T $$