These slides give an introduction to ROC curves.
1. Hello and welcome to this lecture. In this lecture I will present the Receive Operating Characteristics Curve.
2. The receiver operating characteristics or short ROC-Curve was first developed and applied in world war II for detecting enemy objects like submarines in battlefields. Later this technique was applied in psychology, medicine, forecasting of natural hazards, and also for model performance assessment.
3. In the last lecture you have learned what the confusion matrix is. I will explain to you how to get from confusion matrix to ROC curve. In the graph you can see horizontally the actual target value information. It is a binary classification, in which we have the classes 0 and 1. Vertically, you can see the predictions. Prediction of machine learning algorithms usually provide probabilities, so the values range from zero to one. Probabilities above a defined threshold are predicted as class 1, probabilities below the threshold are predicted as class 0. You can see the areas of the confusion matrix, True negatives, false positives, true positives and false negatives. For a given threshold you get a confusion matrix. You can see for the given threshold, the corresponding confusion matrix on the right side.
4. If you adapt the threshold you get an updated confusion matrix. You can see that we increased the threshold. Now the count of true negatives and false negatives is increased, and the count of false positives and true positives and much smaller.
5. Here you see again our confusion matrix. We spoke earlier about true positive rate. It is the ratio of true positives and the sum of true positives and false negatives. This will be our vertical axis in the ROC curve. The area of the affected cells is highlighted in orange.
6. The false positive rate is used as the horizontal axis in the ROC curve. Again, the area of affected cells is highlighted in orange. It is calculated as ratio of false positives and the sum of false positives and true negatives.
7. The threshold is varied from 0 to 1. For all iterations the count of the cells in the confusion matrix is measured, and true positive rate, and false positive rate are calculated. You can see an example table. This information will then be visualised.
8. The ROC curve has false positive rate as X axis, and true positive rate as Y axis. The perfect classifier has zero false positive rate, and a true positive rate of 1. You find it at the upper left corner of this plot. The red, dashed, diagonal line represents a random guess. Better classifiers will be shown as curves closer to the upper left corner, worse classifiers are closer to the random guess line.
9. Now, the ROC curve is calculated and plotted for different models. This is one of the purposes of the ROC curve. With it, you can compare different models. Here you can see for a given task, the ROC curve for logistic regression in red, random forest in green, and support vector machines in blue. All techniques will be presented in separate section. Here, random forest is for all measured combinations of false positive rate and true positive rate, better than the other two models. You can see this, because its curve is always closer to the upper left corner.
10. Comparing different curves is a manual process. If you want to derive one specific measure, you calculate a measure called “Area under Curve”, or short AUC. It maps the complete ROC curve to one measure. This way it is even easier to compare different models. It does exactly what you expect – it calculates the area under the curve. An example you can see at the right side. The sum of the green squares are taken into account as well as half of the blue squares.
11. AUC might not always be the best choice for selecting a model. The reason is that false negatives and false positives are not similarly critical. In the beginning I told you that ROC curve was applied in world war II for detection of enemy submarines. In this example a false negative means there was actually an attack, but the model did not predict it. You can imagine that false negatives are much more critical than false positive. If you only use Area under curve for model selection, this might be misleading. Take a look at the example. You see the ROC curve for two different models for detecting submarines: model A and model B. Both have very similar AUC values. Model A’s AUC is slightly higher, so based on AUC you would choose this model. But if your cost of false negative and false positive is very different, you should take this into account. You can add to the model that false negatives are for example ten times more costly than false positives.
12. You can see two different ROC curves. The colors correspond to different cost parameters. In the left plot, false negatives are treated as much more costly than false positives. The best model has minimum cost related to it. It moves to high values of false positive rate and true positive rate. In the right plot, false positives are considered as much more costly. Now the best model is found for low values of false positive rate and true positive rate.
2. The receiver operating characteristics or short ROC-Curve was first developed and applied in world war II for detecting enemy objects like submarines in battlefields. Later this technique was applied in psychology, medicine, forecasting of natural hazards, and also for model performance assessment.
3. In the last lecture you have learned what the confusion matrix is. I will explain to you how to get from confusion matrix to ROC curve. In the graph you can see horizontally the actual target value information. It is a binary classification, in which we have the classes 0 and 1. Vertically, you can see the predictions. Prediction of machine learning algorithms usually provide probabilities, so the values range from zero to one. Probabilities above a defined threshold are predicted as class 1, probabilities below the threshold are predicted as class 0. You can see the areas of the confusion matrix, True negatives, false positives, true positives and false negatives. For a given threshold you get a confusion matrix. You can see for the given threshold, the corresponding confusion matrix on the right side.
4. If you adapt the threshold you get an updated confusion matrix. You can see that we increased the threshold. Now the count of true negatives and false negatives is increased, and the count of false positives and true positives and much smaller.
5. Here you see again our confusion matrix. We spoke earlier about true positive rate. It is the ratio of true positives and the sum of true positives and false negatives. This will be our vertical axis in the ROC curve. The area of the affected cells is highlighted in orange.
6. The false positive rate is used as the horizontal axis in the ROC curve. Again, the area of affected cells is highlighted in orange. It is calculated as ratio of false positives and the sum of false positives and true negatives.
7. The threshold is varied from 0 to 1. For all iterations the count of the cells in the confusion matrix is measured, and true positive rate, and false positive rate are calculated. You can see an example table. This information will then be visualised.
8. The ROC curve has false positive rate as X axis, and true positive rate as Y axis. The perfect classifier has zero false positive rate, and a true positive rate of 1. You find it at the upper left corner of this plot. The red, dashed, diagonal line represents a random guess. Better classifiers will be shown as curves closer to the upper left corner, worse classifiers are closer to the random guess line.
9. Now, the ROC curve is calculated and plotted for different models. This is one of the purposes of the ROC curve. With it, you can compare different models. Here you can see for a given task, the ROC curve for logistic regression in red, random forest in green, and support vector machines in blue. All techniques will be presented in separate section. Here, random forest is for all measured combinations of false positive rate and true positive rate, better than the other two models. You can see this, because its curve is always closer to the upper left corner.
10. Comparing different curves is a manual process. If you want to derive one specific measure, you calculate a measure called “Area under Curve”, or short AUC. It maps the complete ROC curve to one measure. This way it is even easier to compare different models. It does exactly what you expect – it calculates the area under the curve. An example you can see at the right side. The sum of the green squares are taken into account as well as half of the blue squares.
11. AUC might not always be the best choice for selecting a model. The reason is that false negatives and false positives are not similarly critical. In the beginning I told you that ROC curve was applied in world war II for detection of enemy submarines. In this example a false negative means there was actually an attack, but the model did not predict it. You can imagine that false negatives are much more critical than false positive. If you only use Area under curve for model selection, this might be misleading. Take a look at the example. You see the ROC curve for two different models for detecting submarines: model A and model B. Both have very similar AUC values. Model A’s AUC is slightly higher, so based on AUC you would choose this model. But if your cost of false negative and false positive is very different, you should take this into account. You can add to the model that false negatives are for example ten times more costly than false positives.
12. You can see two different ROC curves. The colors correspond to different cost parameters. In the left plot, false negatives are treated as much more costly than false positives. The best model has minimum cost related to it. It moves to high values of false positive rate and true positive rate. In the right plot, false positives are considered as much more costly. Now the best model is found for low values of false positive rate and true positive rate.