In this interactive dashboard you will learn how to use the ROC curve and how the ROC curve is created based on confusion matrix.
The underlying task is to create a model for a binary classifier. There are two classes: zero and one. At the right side you see the model predictions as the vertical axis, and the actual values as the horizontal axis. This classifier provides predictions, which range from zero to one. It depends on a threshold to assign specific values to the groups: false positive FP, true negative TN, true positive TP, and false negative FN. Points that are actual zero and below the threshold are assigned to TN, above the threshold FP. Points that are actual one and below the threshold are assigned to FN, above the threshold to TP.
We start with a low threshold, for example 0.1. You can see that there is a large group of FP and TP. If we increase the threshold we assign more points to true negative and false negative. The count of points for FP and TP is reduced. At the left side you can see the confusion matrix. The values correspond to the count of points in these four areas.
You can also see the accuracy as one main performance metric. By changing the threshold you change the point assignments, which directly impacts the confusion matrix, and finally the performance metric, which relies on the confusion matrix.
In the below graph you see the ROC curve. True positive ratio is plotted versus False positive ratio. By changing the threshold the model is changed. The red dot corresponds to the current model. There are as many model results as there are thresholds. For each threshold a different model result is created. What happens if you run this algorithm for a complete range of thresholds from zero to one? Right, you get the Receiver Operating Characteristics Curve. You can see it, if you click on “Show Overall ROC curve”. The shape of the curve depends on the algorithm and the corresponding parameters. If now a different model would be fitted, you could compare both and see if one is generally performing better than the other. Here only one curve is shown.
Why do we need cost parameters? Think about when we spoke about Type I and Type II errors. The main idea here was, that certain errors are more costly than the others. For example a false warning of an earthquake might be an annoyance, but not predicting the earthquake would result in huge costs. In cases like this, when you have errors that are not equally critical, it is useful to incorporate the different costs in your model.
I did this in the plot. Now the colors of the second plot become important. They represent a cost value. You can see their explanation in the legend. What you want to find is the minimum cost of a model. In our case the purple colors represent the lowest cost values, and the black point represents the model with the minimum cost.
If we increase our false negatives cost parameter and make it 10 times more costly than false positive costs, this is our earthquake example in which we want to ensure that missing an actual earthquake is much more costly than a false warning. The minimum cost model changes to high values of false positive ratio and true positive ratio. Contrary, if we make false positives 10 times more costly than false negatives, the minimum cost model is found at low false positive ratio and low true positive ratio.
Let’s wrap up what we covered. In this interactive lecture you learned how the threshold influences the confusion matrix and performance metric. You learned that the ROC curve is a combination of a complete range of thresholds. Finally, you learned how treating false negative and false positive costs differently, affects the model selection.
That’s it for this lecture. Thank you very much and see you in the next one.