Esperado Calibração Erro (ECE) is a metric used to evaluate the calibration of previsões probabilísticas feitas por aprendizado de máquina models. Calibration refers to the agreement between predicted probabilities and actual outcomes. In simpler terms, if a model predicts a certain event with a probability of 70%, we would expect that event to occur approximately 70% of the time over a large number of predictions.
ECE quantifies this calibration by comparing the predicted probabilities of a model to the actual frequencies of the events. It is calculated by dividing the probability space into bins and measuring the average difference between the predicted probabilities and the observed outcomes across these bins. A lower ECE indicates better calibration, meaning the model’s predictions are more reliable and trustworthy.
Para calcular o ECE, siga estes passos:
- Agrupe as previsões em intervalos com base nos seus valores de probabilidade previstos.
- Para cada intervalo, calcule o accuracy (a proporção de previsões corretas) e a probabilidade prevista média.
- Calcule a diferença absoluta entre as probabilidades previstas e as precisões reais de cada intervalo.
- Faça a média dessas diferenças em todos os intervalos para obter o ECE.
For practical applications, ECE can be particularly important in fields such as healthcare, finance, and sistemas autônomos, where decision-making relies heavily on the reliability of probability estimates. A well-calibrated model helps stakeholders trust the predictions, leading to better decision-making and risk management.