Calibration Plot
A calibration plot, also known as a reliability diagram, is a graphical representation used to evaluate how well the predicted probabilities from a statistical model align with the actual outcomes. This is particularly important in fields like machine learning and statistics, where models often provide probabilistic predictions.
In a typical calibration plot, the x-axis represents the predicted probabilities (often binned into intervals), while the y-axis shows the actual observed frequencies of the outcomes. For instance, if a model predicts a 70% chance of an event occurring, the calibration plot will show how often that event actually occurs when the model makes such a prediction.
A perfectly calibrated model would result in a calibration plot where the predicted probabilities match the actual outcomes, forming a 45-degree diagonal line (often referred to as the ‘calibration line’). If the plot falls above this line, it indicates that the model is under-confident (predicting lower probabilities than observed). Conversely, if it falls below, the model is over-confident (predicting higher probabilities than observed).
Calibration plots are crucial for assessing models in binary classification tasks, such as in healthcare for predicting disease presence or in finance for credit scoring. By visually inspecting the calibration plot, practitioners can make informed decisions about model adjustments, threshold settings, and overall reliability of the predictions.