Esperado Calibración Error (ECE) is a metric used to evaluate the calibration of predicciones probabilísticas realizado por aprendizaje automático models. Calibration refers to the agreement between predicted probabilities and actual outcomes. In simpler terms, if a model predicts a certain event with a probability of 70%, we would expect that event to occur approximately 70% of the time over a large number of predictions.
ECE quantifies this calibration by comparing the predicted probabilities of a model to the actual frequencies of the events. It is calculated by dividing the probability space into bins and measuring the average difference between the predicted probabilities and the observed outcomes across these bins. A lower ECE indicates better calibration, meaning the model’s predictions are more reliable and trustworthy.
Para calcular el ECE, sigue estos pasos:
- Agrupa las predicciones en intervalos según sus valores de probabilidad predicha.
- Para cada intervalo, calcule el accuracy (la proporción de predicciones correctas) y la probabilidad predicha promedio.
- Calcula la diferencia absoluta entre las probabilidades predichas y las precisiones reales para cada intervalo.
- Promedia estas diferencias en todos los intervalos para obtener el ECE.
For practical applications, ECE can be particularly important in fields such as healthcare, finance, and sistemas autónomos, where decision-making relies heavily on the reliability of probability estimates. A well-calibrated model helps stakeholders trust the predictions, leading to better decision-making and risk management.