Attendu Étalonnage Erreur (ECE) is a metric used to evaluate the calibration of prédictions probabilistes créé par apprentissage automatique models. Calibration refers to the agreement between predicted probabilities and actual outcomes. In simpler terms, if a model predicts a certain event with a probability of 70%, we would expect that event to occur approximately 70% of the time over a large number of predictions.
ECE quantifies this calibration by comparing the predicted probabilities of a model to the actual frequencies of the events. It is calculated by dividing the probability space into bins and measuring the average difference between the predicted probabilities and the observed outcomes across these bins. A lower ECE indicates better calibration, meaning the model’s predictions are more reliable and trustworthy.
Pour calculer l'ECE, suivez ces étapes :
- Regroupez les prédictions en intervalles en fonction de leurs valeurs de probabilité prédites.
- Pour chaque intervalle, calculez le accuracy (la proportion de prédictions correctes) et la probabilité prédite moyenne.
- Calculez la différence absolue entre les probabilités prédites et les précisions réelles pour chaque intervalle.
- Faites la moyenne de ces différences sur tous les intervalles pour obtenir l'ECE.
For practical applications, ECE can be particularly important in fields such as healthcare, finance, and systèmes autonomes, where decision-making relies heavily on the reliability of probability estimates. A well-calibrated model helps stakeholders trust the predictions, leading to better decision-making and risk management.