In the last class we discussed implementing convolutional neural networks for classification problems. However, our discussion of how to properly evaluate classification problems has been limited. Evaluating classification problems for us is important for two reasons. First, it helps us understand fundamental concepts which will also be used when discussing the evaluation of other problems, such as segmentation and detection. Second, because we are in an industry where great costs can be associated with small mistakes we need to be rigorous and comprehensive in our evaluation.

Important notes

1. It is fine to consult with colleagues to solve the problems, in fact it is encouraged.
2. Please turn off AI tools, we want you to memorize concepts and not just quickly breeze through problems. To turn off AI click on the gear in the top right corner. got to AI assistance -> Untick Show AI powered inline completions, Untick consented to use generative AI features, tick Hide Generative AI features

Additional Note

This notebook is designed to be a cheat sheet. If you ever have to discuss with a machine learning developer or researcher about classification problems and their evaluation. You can use this notebook to quickly look up how evaluation for your specific classification problem works!

Lesson 7: Evaluating Classification Problems

In this class we will start with the simplest classification problem: "Binary Classification". In the case of binary classification, there are only two classes; 0 and 1. We will start with basic metrics, their definition, and their interpretation. We will also discuss limitations of these metrics.

Then, we will continue to discus evaluation in the case of class imbalance. What if instead of your dataset being divided into 50/50 positive and negative classes, you instead of 1/99?

After discussing class imbalance we will briefly discuss metrics where a different cost is associated with getting things right or wrong.

Next, we will generalize the concepts we have learned from binary classification to multi-class classification. We will then discuss a less often used case where labels are no longer mutually exclusive. This is called multi-label classification.

Finally, we will discuss another concept: model calibration. Model calibration basically tells you how reliable your predicted probabilities are. For example, if you look at the weather forecast and it says there is an 80% chance of rain, is there an actual 80% chance of rain? In the case of weather forecasts for the next day this is actually pretty close.

At the end, we will talk through some case studies!

7.1 Binary Classification With no Class Imbalance

If you recall the previous tuturials, so far we've only looked at loss curves and occasionally the accuracy. These are fairly basic evaluation metrics, but there are many more.

In general it is best practise to always include a variant (irrespective of whether you are looking at binary or multi-class problems) of these in your analysis.

Confusion Matrix Metrics

Here are some key metrics used to evaluate the performance of a binary classification model:

• True Positives (TP): The number of instances that are actually positive and are correctly predicted as positive.
• False Positives (FP): The number of instances that are actually negative but are incorrectly predicted as positive. This is also known as a Type I error.
• False Negatives (FN): The number of instances that are actually positive but are incorrectly predicted as negative. This is also known as a Type II error.
• True Negatives (TN): The number of instances that are actually negative and are correctly predicted as negative.

These are used to calculate our classification performance metrics.

Here is an image that may make th is more easy to remember:

Classification Performance Metrics

• Accuracy: The proportion of total instances that were correctly predicted (both positive and negative). This is the first evaluation metric that we have already used. It works well enough in case of balanced classes. However, it can be misleading in case of highly imbalanced classes. For example, if you have 99/1 negative to postive examples and your model predicts negative for everything. You would have an accuracy of 99%. This may look impressive, but it really does not tell you much in isolation.

  $$ \text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} $$

• Precision (Positive Predictive Value): The proportion of positive predictions that were actually correctly predicted. It measures the accuracy of the positive predictions.

  $$ \text{Precision} = \frac{TP}{TP + FP} $$

• Recall (Sensitivity, True Positive Rate): The proportion of actual positive instances that were correctly identified. It measures the model's ability to find all positive instances, relative to correctly identified positive and correctly identified negative instances.

  $$ \text{Recall} = \frac{TP}{TP + FN} $$

• Specificity (True Negative Rate): The proportion of actual negative instances that were correctly identified as negative. It measures the model's ability to correctly identify negative instances. We will run into this when we discuss class imbalance, it's not implemented below as such.

  $$ \text{Specificity} = \frac{TN}{TN + FP} $$

• F1 Score (Dice Coefficient): The harmonic mean of Precision and Recall. It provides a single score that balances both metrics. It's particularly useful when you need to balance precision and recall, especially with uneven class distribution. In image analysis, it's often referred to as the Dice Coefficient (as such this is also one of the main metrics in the segmentation tutorial) and calculated as:

  $$ \text{F1 Score} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} = \frac{2 \times TP}{2 \times TP + FP + FN} $$

• Jaccard Index (Intersection over Union - IoU): The ratio of the intersection of the predicted and actual positive sets to their union. Commonly used in image segmentation and object detection. For binary classification, it can be related to the F1 score.

  $$ \text{Jaccard Index (IoU)} = \frac{TP}{TP + FP + FN} $$

Note: $\text{F1} = \frac{2 \times \text{IoU}}{1 + \text{IoU}}$ and $\text{IoU} = \frac{\text{F1}}{2 - \text{F1}}$

There are more metrics, though these are the ones that you will run into most often and are the most important. If you are interested in more metrics, these can be found on this page. You may run into an application where another niche metric is important, so remember that this page exists!

Below we have an example implementation of all of the scores on a simple problem.

The confusion matrix

After calculating the confusion matrix metrics you can plot what is called a confusion matrix. A confusion matrix is a plot by which you can easily see how well your model is doing. The more items are on the main diagonal from the top left to the bottom right, the better your score is. Any off diagonal items are misclassified.

Area under the receiver operating characteristic curve (AUC-ROC)

History of AUC-ROC

The Area Under the Receiver Operating Characteristic (AUC-ROC) curve is a performance metric used for classification problems. Its history dates back to the 1940s, originating in electrical engineering for analyzing radar signals during World War II.

Initially, ROC curves were used to assess the ability of radar receivers to distinguish between signal and noise. The x-axis represented the false positive rate, and the y-axis represented the true positive rate. The curve illustrated the trade-off between these two rates as the detection threshold was varied.

In the 1970s and 1980s, ROC analysis found its way into psychology and medical diagnostics. It became a standard tool for evaluating the performance of diagnostic tests, where the "signal" was the presence of a disease and the "noise" was its absence.

The term "Area Under the Curve" (AUC) emerged as a single scalar value to summarize the overall performance of a classifier across all possible thresholds. An AUC of 1.0 represents a perfect classifier, while an AUC of 0.5 indicates a classifier that performs no better than random chance.

In machine learning, AUC-ROC is widely used to evaluate the performance of binary classifiers, especially when dealing with imbalanced datasets. It provides a comprehensive measure of a model's ability to discriminate between positive and negative classes.

How AUC-ROC Works

The Area Under the Receiver Operating Characteristic (AUC-ROC) curve is a way to evaluate the performance of a binary classifier. To understand how it works, let's break down the components:

1. ROC Curve: The ROC curve is a plot of the True Positive Rate (TPR) against the False Positive Rate (FPR) at various predicted probability threshold settings.

   • True Positive Rate (TPR): Also known as Sensitivity or Recall, it's the ratio of correctly predicted positive instances to the total number of actual positive instances. $TPR = TP / (TP + FN)$.
   • False Positive Rate (FPR): It's the ratio of incorrectly predicted positive instances to the total number of actual negative instances. $FPR = FP / (FP + TN)$.

2. Decision Thresholds: For a classifier that outputs a probability or score, a decision threshold is used to decide whether an instance is classified as positive or negative. By varying this threshold from 1 to 0, (as we move from left to right on the x-axis) we can generate different pairs of TPR and FPR values. What an ROC curve shows you is basically the tradeoff between the TPR and FPR, depending on at what level of probability you decide that a class is predicted as positive or negative.

3. Plotting the Curve: As you decrease the threshold, you will typically increase both the TPR and FPR. Plotting these pairs of (FPR, TPR) for different thresholds gives you the ROC curve.

   • A perfect classifier would have a ROC curve that goes straight up from (0,0) to (0,1) and then across to (1,1). This means it achieves a 100% TPR with a 0% FPR.
   • A completely random classifier would have a ROC curve that follows the diagonal line from (0,0) to (1,1).

4. Area Under the Curve (AUC): The AUC is the area under the entire ROC curve. It provides a single number that summarizes the classifier's performance across all possible thresholds.

   • An AUC of 1 indicates a perfect classifier.
   • An AUC of 0.5 indicates a classifier that performs no better than random chance.
   • An AUC greater than 0.5 suggests that the classifier has some ability to distinguish between positive and negative classes.

In essence, the AUC-ROC tells you the probability that a randomly chosen positive instance will be ranked higher (assigned a higher score/probability) by the classifier than a randomly chosen negative instance.

When (not) to use the AUC

So when should you use the AUC-ROC and the ROC? Here is when:

1. You have roughly balanced classes (i.e. in case of a binary classification problem you would have 50% positive classes and 50% negative classes). Your AUC-ROC can still look great, even when your classes are highly imbalanced.
2. If all you care about is the relative rank order of probabilities, then you should use the AUC-ROC. The AUC will not tell you anything about the reliability of probabilities (i.e. a prediction of 0.8 means that 80% of the time the model is correct). For the latter you will have to look at model calibration, which is discussed later.
3. When decision thresholds are important to your problem, do not blindly use the AUC. The AUC averages all the value at all decision threshold. If you find that you have to operate in a restricted FPR rang, for example when predicting somebody has a scary disease when in reality they don't, you may want to restrict your AUC calculation to a narrower range. This is called a partial AUC (pAUC).
4. When the costs of false positives and false negatives are very different you should not use the AUC blindly. In this case you are better off using cost sensitive.

Let's see an implementation of the ROC Curve from which the AUC is calculated.

7.2 Binary Classification With Class Imbalance

In the last section we looked at how to evaluate a binary classification problem when you have balanced data. In this section we will look at methods to evaluate binary classification in the case of unbalanced data. This is very important in medical applications, because usually healthy people outnumber sick people. So your dataset is never truly balanced. This will also carry over in the tutorial on image segmentation. Pixels in the background of an image almost always outnumber pixels that belong to the object that we are trying to segment.

The first thing that we want to do is to make sure that we do not rely on the accuracy, for the reason we mentioned before. In addition, we always want to report and critically look at the precision, recall, and F1-score. These metric work because they normalise relative to the total amount of positives or the total amount of predicted positives. So just predicting one class will result in a bad score on these metrics.

Another thing that we can do, is plot the precision-recall curve and calculate the area under its curve. This is what we will discuss next.

Precision-Recall Curve

The Precision-Recall curve is another important evaluation metric for binary classification, especially useful when dealing with imbalanced datasets.

1. What it plots: The Precision-Recall curve plots Precision (on the y-axis) against Recall (on the x-axis) at various probability thresholds.

   • Precision: ($\frac{TP}{TP + FP}$) - The ability of the trained classifier to correctly predict the positive class.
   • Recall: $\frac{TP}{TP + FN}$ - The ability of the classifier to find all the positive samples in the dataset.

2. How it works: Similar to the ROC curve, you vary the decision threshold for classifying an instance as positive. For each threshold, you calculate the precision and recall and plot the point on the graph. The threshold is similarly decremented from 1 to 0 as we move down on the x-axis.

3. Interpretation:

   • A curve that stays closer to the top-right corner indicates better performance, meaning the model achieves high precision and high recall simultaneously.
   • A random classifier would have a Precision-Recall curve that is a horizontal line at a precision equal to the prevalence of the positive class in the dataset.
   • The Area Under the Precision-Recall Curve (AUPRC or AP) is a single metric that summarizes the performance across all thresholds. A higher AUPRC indicates better performance. The AUPRC falls between 0 and 1.
   • In the example below you will see that as the recall approaches 1, the precision approaches the prevalence (total amount of times the positive class occurs in the dataset). This is because at that point it will predict everything as belonging to the positive class, as such the number of true positives equals the actual number of actual positives in the data and the number of false positives equals the number of negatives in the data. Leading the number of positive examples being divided by all the data in our dataset, which equals the prevalence.

4. When to use it: The Precision-Recall curve is particularly informative when the positive class is rare (imbalanced data). In such cases, the ROC curve can be misleading because a high True Negative rate can mask poor performance on the positive class. The Precision-Recall curve, focusing only on the positive class, provides a more realistic picture of the model's ability to identify positive instances without generating too many false positives.

In summary, while the ROC curve assesses the classifier's ability to distinguish between classes generally, the Precision-Recall curve is a better indicator of performance when the positive class is the minority class and correctly identifying positive instances is crucial.

Below you can see a plot of the Precision recall curve. Note that we do not simulate data that is unbalanced in this case. This would be quite computationally expensive.

Less common Metrics

So far we have discussed common metrics that can be used to evaluate unbalanced data. Were you to ever publish an academic paper on a task that involves binary classification, then you would definitely include the above metrics and plots. There are two more metrics that can be used that are a little less common, but nonetheless informative. These are Matthews Correlation Coefficient (MCC), and Balanced Accuracy.

Matthews Correlation Coefficient

The Matthews Correlation Coefficient (MCC) is a single-value evaluation metric for binary classification that takes into account all four values in the confusion matrix: True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN). It is considered a more balanced measure than Accuracy, Precision, or Recall, especially when dealing with imbalanced datasets.

The MCC is essentially a correlation coefficient between the observed and predicted binary classifications. It ranges from -1 to +1, where:

• +1 represents a perfect prediction.
• 0 represents a prediction no better than random chance.
• -1 represents a perfect inverse prediction (always predicting the opposite class).

The formula for MCC is:

$$ \text{MCC} = \frac{TP \times TN - FP \times FN}{\sqrt{(TP + FP)(TP + FN)(TN + FP)(TN + FN)}} $$

One of the key advantages of MCC is that it produces a high score only if the classifier performs well in all four aspects of the confusion matrix (TP, TN, FP, FN) proportionally to the size of the positive and negative classes. This makes it a reliable metric for imbalanced datasets where a high accuracy might be misleading.

---

Why the MCC Might Not Be Reported as Frequently:

Despite its strengths, particularly with imbalanced data, MCC is not always the go-to metric in every scenario. Some reasons for this include:

• Interpretability: While the range of -1 to +1 is clear, the specific value of MCC can sometimes be less intuitive to interpret compared to metrics like Precision or Recall, which have direct interpretations related to false positives and false negatives. Stakeholders who are not deeply technical may find it harder to grasp what an MCC of, say, 0.6 means in practical terms for their problem compared to a precision of 0.8.
• Focus on Specific types of Errors: In some applications, there's a much stronger emphasis on minimizing a particular type of error (e.g., minimizing false positives in a medical diagnosis setting). In such cases, metrics that directly reflect the cost of that specific error (like Precision or a cost-sensitive metric) might be preferred or used in conjunction with MCC.
• Comparisons: When comparing results across different studies or models, it's often easier to use metrics that are universally reported in that domain. If MCC is not commonly used in a specific field, it can be harder to benchmark performance.

However, in fields where a balanced assessment of performance across all aspects of the confusion matrix is critical, especially with imbalanced data, MCC is increasingly recognized and used. Just be careful with non-machine learning stakeholders that may not be as familiar with it.

Balanced Accuracy

Balanced Accuracy is an evaluation metric that addresses the issue of misleading accuracy when dealing with imbalanced datasets. It is defined as the average of recall (sensitivity) and specificity.

$$ \text{Balanced Accuracy} = \frac{\text{Recall} + \text{Specificity}}{2} $$

where:

• Recall (Sensitivity): $\frac{TP}{TP + FN}$ (True Positive Rate)
• Specificity: $\frac{TN}{TN + FP}$ (True Negative Rate)

It is useful to know that specificity has a relationship to the false positive rate. This relationship is $1 - FPR$.

Balanced Accuracy is useful because it gives equal weight to the performance on both the positive and negative classes, regardless of their proportion in the dataset. A model that simply predicts the majority class in an imbalanced dataset will have a high standard accuracy but a balanced accuracy closer to 0.5 (random chance).

---

Why Balanced Accuracy Might Not Be Reported as Frequently:

While a valuable metric for imbalanced data, Balanced Accuracy might not be reported as frequently as some other metrics for several reasons:

• Relationship to ROC AUC: For a binary classifier that outputs scores or probabilities, Balanced Accuracy is equivalent to the Area Under the ROC Curve (AUC-ROC) when the ROC curve is calculated based on a specific set of thresholds (or when averaging over random thresholds). Since AUC-ROC is a very common and well-understood metric, researchers often prefer to report AUC-ROC as it provides a summary across all possible thresholds, whereas Balanced Accuracy is often calculated at a single, optimal threshold or represents an average across a specific thresholding strategy. Remember that ROC is basically a plot with FPR against
• Focus on Specific types of Errors: Similar to the MCC, in applications where the cost of false positives and false negatives are significantly different, metrics like Precision and Recall (or cost-sensitive metrics) might be more directly informative about the type of errors that are most critical to minimize. Balanced Accuracy provides an overall balance but doesn't highlight performance on one specific type of error over another.
• Interpretability Compared to Precision/Recall: While more informative than standard accuracy for imbalanced data, Balanced Accuracy can still be less intuitive to interpret in terms of concrete false positive or false negative rates compared to looking at Precision and Recall separately.

7.3 Cost Sensitive Evaluation

In many real-world applications, the costs associated with different types of classification errors are not equal. For example, in medical diagnosis, a False Negative (failing to detect a disease when it's present) might be significantly more costly than a False Positive (incorrectly diagnosing a healthy person with the disease). Cost-Sensitive Evaluation and the use of a Cost Matrix are methods that take these differential costs into account.

A Cost Matrix is a table that defines the cost associated with each possible outcome of a binary classification:

	Predicted Positive	Predicted Negative
Actual Positive	Cost of True Positive (often 0)	Cost of False Negative (FN)
Actual Negative	Cost of False Positive (FP)	Cost of True Negative (often 0)

The costs in the matrix are typically defined based on the specific problem and its consequences. Using this matrix, you can calculate the Total Cost of the classification model:

$$ \text{Total Cost} = (TP \times \text{Cost}_{TP}) + (FP \times \text{Cost}_{FP}) + (TN \times \text{Cost}_{TN}) + (FN \times \text{Cost}_{FN}) $$

Where $\text{Cost}_{TP}$ is the cost of a True Positive, $\text{Cost}_{FP}$ is the cost of a False Positive, and so on. Often, the costs of True Positives and True Negatives are considered to be 0, but they can be assigned values if there are benefits or costs associated with correct classifications.

Evaluation then focuses on minimizing the Total Cost. Metrics can be derived from this total cost. For instance, instead of maximizing accuracy, you would aim to minimize the average cost per instance.

Using cost-sensitive evaluation is crucial when the impact of different errors varies significantly, providing a more relevant measure of a model's performance in a practical context.

We will now replicate aspects of a tutorial on scikit-learn to show you how this is done. In this tutorial the threshold probability is tuned such that the cost is minimized.

our cost matrix looks like this:

	Predicted Positive	Predicted Negative
Actual Positive	0	3 (FN)
Actual Negative	5 (FP)	0

We will use this to calculate our costs!

So how well does the classifier work Below we plot the ROC, the Precision/recall curve, and the cost curve. On the training set, we see that it seems to work well.

We see that for our cost function, we get this concave plot. Where there is a clear minimum. This is due to us assigning a cost to both true positives and negatives.

Next, we will select a threshold, to minimize the total cost on our testing set. We will keep it simple this time and calculate the cost at each index on the line and select the minimum.

Summary of Results

Based on the analysis using manual threshold optimization, we compared the performance of the logistic regression model on the test set using a default probability threshold of 0.5 and an optimal threshold determined by evaluating costs across a range of thresholds on the test set.

• Default Threshold ($0.5$): The total cost on the test set with the default threshold was $19$. The confusion matrix showed $3$ false negatives and $2$ false positives.
• Optimal Threshold ($0.27$): Using the optimal threshold of 0.27 (found through manual optimization on the training set) on the test set resulted in a total cost of $18$. The confusion matrix for the optimal threshold showed $6$ false negatives and $0$ false positives.

In this specific case, tuning the threshold through manual optimization resulted in a slightly lower total cost on the test set ($18$) compared to the default threshold ($19$). Nothing phenomenal, but still an improvement.

7.4 Multi-class Classification Problems

So far, we’ve focused on binary classification problems, where there are only two possible outcomes (e.g., presence or absence of a disease). However, many real-world scenarios involve classifying instances into more than two mutually exclusive classes. This is known as multi-class classification.

Evaluating multi-class models requires extending the concepts we’ve learned for binary classification. Some metrics, such as accuracy, can be applied directly. Others, like precision, recall, and F1-score, require modifications to handle multiple classes. In this section, we will explore:

1. How to adapt binary metrics for multi-class settings.
2. New metrics designed specifically for multi-class evaluation.
3. Best practices for interpreting these metrics in real-world applications.

Strategies for Multi-Class Evaluation: One-vs-Rest and One-vs-One

When moving from binary to multi-class classification, many evaluation metrics (such as precision, recall, and F1-score) are originally defined for two classes. To apply these metrics in a multi-class setting, we use strategies that break the problem into multiple binary evaluations. The two most common approaches are One-vs-Rest (OvR) and One-vs-One (OvO). Finally, we will discuss the multi-class confusion matrix. Which is a generalization of the common binary confusion matrix.

---

One-vs-Rest (OvR)

When we have more than two classes, like classifying different grades of cancer (Grade 1, Grade 2, Grade 3), we can use the One-vs-Rest (OvR) strategy to evaluate our model. Think of it as breaking down the multi-class problem into several simpler binary problems.

Concept:

For each class, we essentially ask a "yes or no" question: "Is this instance of this specific class, or is it not?"

Here's how it works:

1. Pick one class: Let's say we pick "Cancer Grade 1".
2. Create a binary problem: We treat all instances of "Cancer Grade 1" as the positive class. All other instances (Cancer Grade 2, Cancer Grade 3, and any other classes) are grouped together as the negative class ("Not Cancer Grade 1").
3. Evaluate like binary: We then calculate standard binary classification metrics (like Precision, Recall, and F1-score) for this specific "Cancer Grade 1 vs. Rest" problem. Precision here tells us, "When the model predicted 'Cancer Grade 1', how often was it actually Grade 1?". Recall tells us, "Of all the actual 'Cancer Grade 1' cases, how many did the model correctly identify?".
4. Repeat for all classes: We repeat this process for every other class. So, we'll have a "Cancer Grade 2 vs. Rest" evaluation and a "Cancer Grade 3 vs. Rest" evaluation.

The One-vs-Rest strategy is a straightforward way to extend binary metrics to multi-class problems and provides detailed performance metrics for each individual class, which is very useful in medical applications where identifying a specific disease subtype might be critical. However, a potential drawback is that when some classes have very few instances compared to others (highly imbalanced), the "Rest" category can be dominated by the majority classes, potentially making the evaluation for the minority class less informative. This is a common challenge in medical datasets.

---

One-vs-One (OvO)

Another strategy for extending binary evaluation metrics to multi-class problems is the One-vs-One (OvO) approach. Instead of comparing each class against all others, OvO compares every possible pair of classes.

Concept:

Imagine you have classes A, B, and C. With OvO, you would set up separate evaluations for:

• Class A vs. Class B
• Class A vs. Class C
• Class B vs. Class C

For each pair, you only consider the instances belonging to those two classes and build a binary classification problem. You then calculate binary metrics (like Precision, Recall, F1-score) for each of these pairwise comparisons.

If you have $(k)$ classes, the number of pairwise comparisons you need to perform is $(\frac{k(k-1)}{2})$. For example, with 3 classes (A, B, C), you have $(\frac{3(3-1)}{2} = 3)$ pairs. With 4 classes (A, B, C, D), you have $(\frac{4(4-1)}{2} = 6)$ pairs.

A significant drawback of the One-vs-One strategy is that it can become computationally expensive for a large number of classes $(k)$ because the number of binary evaluations grows quadratically with $(k)$, and interpreting the overall performance based on many pairwise metrics can be more complex than interpreting per-class metrics from OvR. However, the reporting can be very detailed.

---

Why These Matter

By breaking down the multi-class problem into a series of binary problems, we can leverage the well-understood concepts of True Positives, False Positives, True Negatives, and False Negatives to assess performance for each class or each pair of classes. This provides a more granular view of where the model is succeeding or failing beyond just overall accuracy.

Building upon these strategies, we can then calculate multi-class generalizations of key binary metrics. We can compute precision, recall, and F1-score for each class using the One-vs-Rest approach, and then combine these per-class scores using micro-averaging, macro-averaging, or weighted averaging to get overall performance summaries that account for class imbalance in different ways. Similarly, we can extend the concepts of ROC curves and Precision-Recall curves to the multi-class setting, often by plotting curves for each class (OvR) or averaging curves across pairs (OvO), providing visual tools to understand the trade-offs between different error types across varying decision thresholds.

Beyond these adaptations of binary metrics, other evaluation metrics are particularly useful in multi-class and related multi-label scenarios common in healthcare. The multi-class confusion matrix is a direct extension of the binary confusion matrix, showing the counts of actual vs. predicted classes in a table format, allowing for easy identification of which classes are being confused with each other.

---

Multi-Class Confusion Matrix

The OvO and RvO strategies all depend on evaluating a multi-class problem by turning it into multiple single class problem. There is another useful plot that you will see frequently when assessing multi-class. This plot counts how many times each class was correctly predicted (on the main diagonal, these are your TP) and how many times it was incorrectly predicted as another class (off diagonal items). If you exclude the main diagonal, each row contains the False negatives for that class and each column contains the False positives for that class. The better the performance, the more items are on your main diagonal.

Below you will see an example of a confusion matrix on generated data.

Averaging Strategies for Multi-Class Metrics

When performing One-vs-Rest or One-vs-One evaluation, we get performance metrics (like Precision, Recall, F1-score) for each individual class or each pair of classes. While per-class metrics are informative and should always be shown, we often need a single overall score to summarize the model's performance across all classes. This is where averaging strategies come in.

---

Micro-averaging

Micro-averaging is an averaging strategy that aggregates the contributions of all classes to compute the overall average metric. Instead of calculating the metric (like precision or recall) for each class separately and then averaging, micro-averaging is calculated globally. This is done by summing up the individual True Positives (TP), False Positives (FP), and False Negatives (FN) from the confusion matrices of all the per-class binary problems (as in the One-vs-Rest approach).

For example, to calculate the micro-averaged precision, you would sum up the TPs from all classes and divide by the sum of TPs and FPs from all classes:

$$ \text{Micro-averaged Precision} = \frac{\sum_{i=1}^{k} TP_i}{\sum_{i=1}^{k} TP_i + \sum_{i=1}^{k} FP_i} $$

where $(TP_i)$ and $(FP_i)$ are the True Positives and False Positives for class $(i)$, and $(k)$ is the number of classes.

A key characteristic of micro-averaging is that it is heavily influenced by the performance on larger classes. Classes with more instances will contribute more to the sums of TP, FP, and FN, and thus have a greater impact on the final micro-averaged score.

Crucially, for metrics like Precision, Recall, and F1-score, the micro-averaged score is equivalent to the overall accuracy of the multi-class classifier. This is because the sum of TPs across all binary OvR problems equals the total number of correctly classified instances in the multi-class problem, and the sum of TPs, FPs, TNs, and FNs across all binary OvR problems equals the total number of instances, which is the denominator for accuracy. Therefore, micro-averaged F1, precision, and recall all reduce to overall accuracy:

$$ \text{Micro-averaged F1} = \frac{2 \times \text{Total TP}}{\text{Total TP} + \text{Total FP} + \text{Total FN}} = \frac{\text{Total Correct Predictions}}{\text{Total Instances}} = \text{Overall Accuracy} $$

This equivalence means that micro-averaging might not be the most informative metric when dealing with imbalanced datasets, as it mirrors the potential misleading nature of overall accuracy in such cases. It essentially treats every instance prediction equally, regardless of the class it belongs to.

---

Macro-averaging

Macro-averaging is an averaging strategy that calculates the metric independently for each class and then takes the simple average of these per-class metrics. Unlike micro-averaging, macro-averaging gives equal weight to each class, regardless of the number of instances it contains.

Here's how it works:

1. Calculate Metric per Class: Compute the desired metric for each class individually, typically using the One-vs-Rest approach.
2. Simple Average: Sum up the metric values obtained for each class and divide by the total number of classes.

For example, the macro-averaged F1-score is calculated as:

$$ \text{Macro-averaged F1} = \frac{1}{k} \sum_{i=1}^{k} F1_i $$

where $(F1_i)$ is the F1-score for class $(i)$, and $(k)$ is the number of classes.

Key Characteristics:

• Equal Class Contribution: Each class has the same impact on the final macro-averaged score, even if one class has significantly more instances than another.
• Sensitivity to Minority Classes: Macro-averaging is particularly useful when you want to ensure that the model performs well on minority classes. A poor performance on a small class will significantly lower the macro-averaged score, even if the model performs excellently on large classes.
• Not Equivalent to Accuracy: Unlike micro-averaging, macro-averaging is generally not equivalent to overall accuracy, especially in the presence of class imbalance.

Macro-averaging is a valuable metric when your goal is to give equal importance to the correct classification of instances from all classes, highlighting issues with performance on rare classes.

---

Weighted-averaging

Weighted-averaging is a strategy that calculates the metrics independently for each class, similar to macro-averaging, but then takes a weighted average where the weight for each class is proportional to the number of instances in that class (its support). This provides a balance between macro-averaging (equal weight to all classes) and micro-averaging (which implicitly weights by instance count).

Here's how it works:

1. Calculate Metric per Class: Compute the desired metric (e.g., Precision, Recall, F1-score) for each class individually.
2. Calculate Class Support: Determine the number of instances for each class in the dataset (or the relevant subset being evaluated).
3. Weighted Average: Sum up the metric values obtained for each class, multiplied by their respective support, and then divide by the total number of instances.

For example, the weighted-averaged F1-score is calculated as:

$$ \text{Weighted-averaged F1} = \frac{\sum_{i=1}^{k} F1_i \times \text{Support}_i}{\sum_{i=1}^{k} \text{Support}_i} $$

where ($F1_i$) is the F1-score for class ($i$), ($\text{Support}_i$) is the number of instances in class ($i$), and ($k$) is the number of classes.

Key Characteristics:

• Reflects Class Distribution: The weighted average score reflects the model's performance on the dataset as a whole, taking into account the frequency of each class.
• Compromise Metric: It sits between micro-averaging and macro-averaging. It is not as sensitive to poor performance on very small minority classes as macro-averaging is, but it doesn't completely ignore the performance on those classes like micro-averaging can appear to do (since micro average is equivalent to accuracy).
• Often Used for Overall Summary: Weighted-averaging is a common choice for providing a single summary metric for multi-class classification, as it gives a sense of the model's overall performance while still accounting for potential imbalance by weighting by class frequency.

Weighted-averaging is particularly useful when you want an overall metric that reflects the model's performance in proportion to how often each class appears in the data.

---

Example: Generalizing Binary Classification Metrics (Part 1)

Remember the classification metrics that we used for binary classification? We are going to apply those to a multi-class classification problem by using a One-vs-Rest strategy. Then we will discuss various strategies of combining them into a single metric. We will use the Forest cover type dataset from scikit-learn. This is a more difficult dataset to work with which is also imbalanced.

Should you wish to run the code, loading the data will take a minute and training the machine learning model will take a few minutes.

Conclusion

One-vs-Rest Table

Based on the One-vs-Rest evaluation table, we can observe the Random Forest classifier's performance on this imbalanced dataset. The model demonstrates strong performance on the majority classes (Classes 0 and 1, highlighted in green), with high accuracy, precision, recall, and F1-scores.

However, the high accuracy reported for the minority classes (Classes 3 and 4, highlighted in red) can be misleading due to their low prevalence. While accuracy appears high (above 99%), this is largely influenced by correctly identifying the majority of instances as not belonging to these rare classes (True Negatives).

Looking at other metrics provides a more realistic view of performance on these minority classes. Recall (Sensitivity) for Class 3 (85.6%) and Class 4 (77.0%) is notably lower than for majority classes, indicating the model misses a significant portion of actual instances from these groups (higher False Negatives). Precision is also generally lower, meaning when the model predicts these classes, there's a higher chance of being incorrect. Metrics like F1-score, Balanced Accuracy, and MCC, which provide a more balanced assessment than accuracy, also show considerably lower values for the minority classes (e.g., F1-scores of 88.5% and 85.2%), confirming that performance on these rarer classes is not as robust as accuracy alone would suggest. This highlights why it's crucial to examine multiple metrics when dealing with imbalanced data in medical applications, where correctly identifying rare conditions is often critical.

NOTE: the high per-class OvR accuracies for minority classes primarily reflect the model's ability to correctly identify instances that do not belong to that specific rare class (TN), rather than its ability to correctly identify instances that do belong to it (TP). The overall accuracy provides a more realistic picture of the model's performance across the entire multi-class problem, taking into account the model's performance on all classes and the prevalence of each class. Recall the accuracy formula.

Averaged Metrics

The averaged metrics table provides single summary scores for the model's performance across all classes. The Micro Average is equivalent to the overall accuracy (95.3%), reflecting the model's performance on the dataset as a whole, where every instance is equally important. This average is heavily influenced by the performance on the larger, majority classes.

The Macro Average provides a different perspective by giving equal weight to each class, including the smaller minority classes. Comparing the Macro average values to the Micro average (e.g., Macro F1 is 92.4% vs. Micro F1/Accuracy 95.3%) highlights that the model performs less well on the minority classes compared to the majority classes, as poor performance on small classes has a larger impact on the Macro average. This makes the Macro average a more informative metric than Micro average (or overall accuracy) when evaluating performance on imbalanced datasets, as it clearly shows issues with minority classes.

Finally, the Weighted Average provides a balance between the Micro and Macro averages. It considers the performance on each class but weights its contribution by the class's frequency in the dataset. This gives a sense of the model's overall performance while still accounting for the class distribution. The Weighted Average F1 (95.3%) in this case is very close to the Micro Average/Overall Accuracy, which again reflects the dominance of the majority classes in the dataset. Comparing these different averages helps provide a more complete understanding of the model's performance beyond just a single number, especially in the context of imbalanced data.

Confusion Matrix

The confusion matrix shows that the model is effective at classifying the majority classes, but that it makes mistakes when classifying minority classes.

Multi-Class ROC and Precision Recall Curves

We've already seen and discussed the binary cases of the ROC (and the AUC) and the Precision/Recall curve. Once again we can choose the One-vs-Rest (OvR) strategy to create the plots and use an averaging strategy to calculate the AUC for both types of curves. It is also possible to use the OvO strategy, but we've chosen not to show this case in the notebook for the sake of brevity.

Multi-Class ROC Curve and AUC

In the multi-class setting, we can generate an ROC curve for each class using the OvR approach. For each class, we treat it as the positive class and all other classes as the negative class. We then calculate the True Positive Rate (TPR) and False Positive Rate (FPR) at various probability thresholds for this binary problem and plot the ROC curve.

The Area Under the ROC Curve (AUC) can then be calculated for each individual class's OvR curve. To get a single overall metric, we can use the averaging strategies discussed earlier.

The multi-class ROC curve and its AUC are useful for understanding the model's overall ability to discriminate between classes across various thresholds.

Multi-Class Precision-Recall Curve and AUPRC

Similar to the ROC curve, we can generate a Precision-Recall curve for each class using the OvR approach. For each class, we plot Precision against Recall at various probability thresholds.

The Area Under the Precision-Recall Curve (AUPRC or AP) is particularly informative in the multi-class setting, especially when dealing with imbalanced datasets. We can calculate the AUPRC for each individual class's OvR curve. Again, these can be averaged using one of the averaging strategies mentioned earlier.

The multi-class Precision-Recall curve and its AUPRC are preferred over ROC when dealing with imbalanced datasets, as they focus on the model's ability to correctly identify positive instances without generating too many false positives, which is crucial for minority classes.

Have a look at the code below that demonstrates how to plot these curves and calculate the averaged AUC and AUPRC values for the Forest Cover Type dataset. You will see that the precision recall curve more clearly illustrates the more difficult to predict minority classes. Whereas the ROC curve has difficulty differentiating them. Though all scores are very high (it's a toy dataset!).

7.5 Multi-Label Classification Problems

We've learned about binary classification, where an attribute is either present or absent, and we've learned about multi-class classification, where one of many attributes is either present or absent. These previous problems have focussed on cases where classe are mutually exclusive. But not all problems are like this. For example, if you have a chest x-ray, multiple organs are present. If you want to indicate what organs are present the previous problem definitions make no sense, because multiple things can be either present or absent. A problem that fits this type of situation is called a multi-label classification problem. As we will see, there is some carry over from the other problems that we have already discussed, but some additional metrics will have to be introduced.

---

Challenges with multi-label data

Multi-label classification presents several unique challenges compared to binary or multi-class problems, primarily due to the possibility of multiple labels being associated with a single instance and the relationships between these labels.

• Label Dependence: A significant challenge is that labels are often not independent of each other. For example, in medical image analysis, inserting a device that is visible on an image may in most cases be associated with the presence of an illness. Think about the presence of a tube on an X-Ray image that is used during intubation of a patient. All patients that are intubated may have an pneumonia, but not all patients that have pneumonia are intubated. Many traditional multi-label models implicitly assume label independence, which can limit their ability to capture these complex relationships and lead to suboptimal performance, as they might miss important patterns or predict improbable combinations of labels.
• Data Imbalance: Similar to multi-class problems, multi-label datasets frequently suffer from data imbalance, but it's at the label level. Some labels appear much more frequently across the dataset than others. Training models to perform well on rare labels is difficult because there are limited examples to learn from. This can be particularly problematic in domains like healthcare, where rare diseases or conditions are often the most critical to identify correctly.
• Evaluation Complexity: Evaluating the performance of multi-label models is more complex than evaluating binary or multi-class classifiers. A prediction for a single instance can be partially correct (some labels predicted correctly, others missed or falsely included). Traditional metrics and confusion matrices don't easily capture this nuance.
• Threshold Selection: Unlike binary or multi-class problems where a single decision boundary or threshold is often sufficient for prediction, multi-label classification typically involves predicting probability scores for each label independently. Converting these scores into binary predictions requires setting a threshold for each label. Choosing the right threshold(s) is crucial as it directly impacts the trade-off between false positives and false negatives for each label and can significantly affect overall performance, especially when dealing with varying label frequencies or asymmetric costs of errors. Finding optimal thresholds, potentially different for each label, adds another layer of complexity to model deployment and evaluation.

Power Set Transformation

One intuitive way to approach multi-label classification is by transforming it into a multi-class classification problem. This can be done using the Power Set Transformation.

The power set of a set $S$ is the set of all subsets of $S$, including the empty set and the set $S$ itself. For example, if we have a set of labels $L = \{L_1, L_2, L_3\}$, the power set $\mathcal{P}(L)$ would be:

$$ \mathcal{P}(L) = \{\emptyset, \{L_1\}, \{L_2\}, \{L_3\}, \{L_1, L_2\}, \{L_1, L_3\}, \{L_2, L_3\}, \{L_1, L_2, L_3\}\} $$

In the context of multi-label classification, the power set transformation treats each unique combination of labels observed in the dataset as a single, distinct class. If the original labels are $\{L_1, L_2, \dots, L_k\}$, the power set transformation creates a new set of classes corresponding to every possible subset of these labels that appears in the training data.

For an instance with true labels $\{L_1, L_3\}$, instead of predicting '1' for $L_1$ and $L_3$ and '0' for $L_2$, the model would be trained to predict the single class corresponding to the label combination $\{L_1, L_3\}$. The problem is converted into a standard multi-class classification task where the classes are these unique label combinations.

Impracticality of Power Set Transformation

While conceptually simple, the power set transformation is frequently impractical for real-world multi-label problems due to a major limitation: the number of possible unique label combinations (classes) can be extremely large.

• Exponential Growth: If there are $k$ possible labels, the total number of subsets in the power set is $2^k$. While the number of unique combinations observed in the training data might be less than $2^k$, it can still grow exponentially with the number of labels. For example, with 20 labels, $2^{20}$ is over a million potential combinations.
• Sparse Data: Even if the total number of possible combinations is large, many combinations might never appear in the training data. This leads to a multi-class problem with a vast number of classes, most of which have zero training examples. Training a classifier to predict classes it has never seen is impossible.
• Increased Model Complexity: Training a multi-class classifier with a huge number of output classes requires a much more complex model and significantly more training data to learn to distinguish between all these combinations.

Due to these scalability and data sparsity issues, the power set transformation is typically only feasible for multi-label problems with a small number of labels. If it is possible for your problem to do this you can simply use multi-class evaluation. However, this section of the tutorial will however focus on the case where you cannot use the power-set transformation.

Instance-Based Metrics

Instance-based metrics evaluate the performance of a multi-label classifier on a per-instance basis. An instance here refers to all the labels associated with a datapoint. For each instance in the dataset, these metrics compare the predicted set of labels directly to the true set of labels. A score is calculated for each individual instance, and the final metric is typically the average of these scores across all instances in the dataset.

An instance-based approach provides insight into how well the model performs for each individual data point, considering the specific combination of labels associated with it. Common instance-based metrics include Exact Match Ratio (Subset Accuracy), Hamming Loss, and the Jaccard Index (averaged over instances). They offer a perspective on how often the model gets the entire set of labels right for a single instance or the average overlap between the predicted and true label sets per instance.

Exact Match Ratio (Subset Accuracy)

The Exact Match Ratio, also known as Subset Accuracy, is one of the strictest multi-label evaluation metrics. It is the proportion of instances for which the set of predicted labels exactly matches the set of true labels. For each instance, if the set of predicted labels is identical to the set of true labels, the instance is counted as a "full match". The Exact Match Ratio is the total count of full matches divided by the total number of instances:

$$ \text{Exact Match Ratio} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{I}(Y_i = \hat{Y}_i) $$

where $N$ is the number of instances, $Y_i$ is the set of true labels for instance $i$, $\hat{Y}_i$ is the set of predicted labels for instance $i$, and $\mathbb{I}(\cdot)$ is the indicator function (which is 1 if the condition is true, and 0 otherwise). This metric provides a simple, intuitive understanding of how often the model gets the entire set of labels right for an instance. It is a very strict metric because even a single missing or incorrectly added label for an instance results in a score of 0 for that instance. A higher Exact Match Ratio indicates better performance (ranging from 0 to 1).

Hamming Loss

The Hamming Loss measures the average number of incorrect labels per instance or, equivalently, the fraction of labels that are incorrectly predicted across all instances and all labels.

Definition: It is the average symmetric difference between the true and predicted label sets for each instance. This means it counts both false positives (predicted label is not a true label) and false negatives (true label is not a predicted label).

Calculation: For each instance, the Hamming Loss is the number of labels incorrectly predicted (either predicted as present but are absent, or predicted as absent but are present), divided by the total number of possible labels. The overall Hamming Loss is the average of these per-instance losses.

$$ \text{Hamming Loss} = \frac{1}{N \times L} \sum_{i=1}^{N} \sum_{j=1}^{L} \mathbb{I}(y_{ij} \neq \hat{y}_{ij}) $$

where $N$ is the number of instances, $L$ is the total number of possible labels, $y_{ij}$ is the true label (1 if present, 0 if absent) for instance $i$ and label $j$, and $\hat{y}_{ij}$ is the predicted label.

Interpretation: A lower Hamming Loss indicates better performance. It ranges from 0 (perfect prediction for all labels on all instances) to 1 (every label is incorrectly predicted for every instance).

Jaccard Index (Jaccard Similarity, IoU)

We've mentioned the jaccard index before under binary classification. However, we can also adapt it to multi-label classification. This is because we would like to see the "overlap" between the label and the prediction.

In binary classification, you would calculate it over the entire dataset. In the case of multi-label classification you calculate it for each instance (datapoint). Then you macro-average it to summarize it over your entire dataset.

$$ \text{Jaccard Index for instance } i = \frac{|Y_i \cap \hat{Y}_i|}{|Y_i \cup \hat{Y}_i|} $$

where $Y_i$ is the set of true labels for instance $i$, and $\hat{Y}_i$ is the set of predicted labels for instance $i$.

Interpretation: A higher Jaccard Index indicates better performance, meaning the predicted set of labels has more overlap to the true set of labels for each instance or globally across the dataset. It ranges from 0 (no overlap) to 1 (perfect match).

Label-Based Metrics

Label-based metrics evaluate the performance of a multi-label classifier on a per-label basis. Instead of looking at the entire set of labels for a single instance, this approach treats the multi-label problem as a collection of independent binary classification problems, one for each label. For each label, we calculate standard binary classification metrics (like Precision, Recall, and F1-score) by considering all instances in the dataset and whether that specific label is present or absent, and whether the model correctly predicted its presence or absence.

We can then use the previously discussed averaging strategies to get a single score for our models performance. Label-based metrics are particularly useful for understanding how well the model performs on individual labels, especially in the presence of label imbalance.

Why is IoU not treated under label-based metrics

Although you could calculate a Jaccard Index for each label across all instances (similar to label-based Precision or Recall), the Jaccard Index (IoU) is conventionally categorized as an instance-based metric in multi-label classification. This is because its most common and intuitive definition is applied at the instance level, measuring the overlap between the set of predicted labels and the set of true labels for each individual data point. The overall IoU is then typically calculated by averaging these per-instance scores. This contrasts with standard label-based metrics like Precision, Recall, and F1, which are primarily calculated by aggregating True Positives, False Positives, and False Negatives per label across all instances and then averaging these per-label scores.

Precision, Recall, and F1-score (Label-Based)

Precision, Recall, and F1-score are adapted for multi-label classification by calculating True Positives (TPs), False Positives (FPs), and False Negatives (FNs) on a per-label basis. For each label, we treat the problem as a binary classification task (presence or absence of that specific label) and calculate the binary metrics.

Let's take Precision as an example. To calculate the precision for a specific label, say Label $j$:

1. We consider all instances in the dataset.

2. For each instance, we check if the true label is $j$ and if the model predicted label $j$.

3. We count the total number of times Label $j$ was truly present and predicted by the model across all instances. This is the True Positives ($TP_j$) for Label $j$.

4. We count the total number of times Label $j$ was not truly present but was predicted by the model across all instances. This is the False Positives ($FP_j$) for Label $j$.

5. The Precision for Label $j$ is then calculated using the standard binary formula:

   $$ \text{Precision}_j = \frac{TP_j}{TP_j + FP_j} $$

   This tells us, out of all the times the model predicted Label $j$, how often it was actually correct.

The rest of the label-based metrics, such as Recall,Specificity or Balanced Accuracy, can be calculated analogously for each label by determining the per-label TP, FP, TN, and FN counts across all instances and applying the corresponding binary metric formula.

Multi-Label Confusion Matrix, ROC, and Precision-Recall Curves

While a single standard multi-class confusion matrix doesn't directly apply to multi-label classification (as instances can have multiple true and predicted labels), we can adapt the concept to gain insight into performance per label. Similarly, ROC and Precision-Recall curves, along with their AUCs, can be generated on a per-label basis.

• Confusion Matrix per Label: For each label, we can construct a binary confusion matrix by treating the problem as predicting the presence or absence of that specific label across all instances. This matrix will show the True Positives, False Positives, False Negatives, and True Negatives for that label. Displaying these per-label confusion matrices provides a detailed view of where the model is succeeding or failing for each individual label.

• ROC Curve and AUC per Label: Similar to the multi-class One-vs-Rest approach, we can generate a binary ROC curve for each label. This plots the True Positive Rate (Recall) against the False Positive Rate at various probability thresholds for predicting that specific label. The Area Under the ROC Curve (AUC) can be calculated for each label's curve. Averaging these per-label AUCs (micro, macro, or weighted) provides overall performance summaries.

• Precision-Recall Curve and AUPRC per Label: Also using a per-label binary approach, we can plot the Precision against the Recall at various probability thresholds for each label. The Area Under the Precision-Recall Curve (AUPRC or AP) for each label is particularly informative for imbalanced multi-label datasets, highlighting the trade-off between false positives and false negatives when identifying instances of that specific label. Averaging these per-label AUPRCs is a common way to summarize performance.

These per-label visualizations and metrics are important for understanding the model's performance on individual labels, especially in datasets with label imbalance, and they complement the instance-based metrics by providing a different perspective on evaluation.

Example: Simulated Multi-Label Classification

In the following example, we give an example of how multi-label classification works. We do the following:

• Simulate data
• Show why the powerset transformation will not always lead to good outcomes.
• Train classifiers for multi-label classification.
• Evaluate them on instance-based and example-based metrics. We will forego the ROC, Precision/recall plots and confusion matrices.

We mentioned the power set transformation before. This plot below will illustrate the difficulty with this. Run the cell above and below. You will see that some of the combinations have few datapoints.

This makes it hard to train and evaluate a machine learning model on these classes. Ofcourse this is a bit of an artefact of the simulation process, but it proves the point.

Next, we are going to train a logistic regressor for each of the labels. With other classifiers there are more sophisticated ways but for our illustration, this will suffice. Then we will show instance-based and label-based evaluation.