期待値 キャリブレーション 誤差（ECE） is a metric used to evaluate the calibration of 予測 機械学習によって作られた models. Calibration refers to the agreement between predicted probabilities and actual outcomes. In simpler terms, if a model predicts a certain event with a probability of 70%, we would expect that event to occur approximately 70% of the time over a large number of predictions.
ECE quantifies this calibration by comparing the predicted probabilities of a model to the actual frequencies of the events. It is calculated by dividing the probability space into bins and measuring the average difference between the predicted probabilities and the observed outcomes across these bins. A lower ECE indicates better calibration, meaning the model’s predictions are more reliable and trustworthy.
ECEを計算するには、次の手順に従います。
- 予測を、その予測確率値に基づいてビンに分類します。
- 各ビンについて、計算します accuracy （正解予測の割合）と平均予測確率。
- 各ビンの予測確率と実際の正確さとの差の絶対値を計算します。
- これらの差をすべてのビンで平均し、ECEを得ます。
For practical applications, ECE can be particularly important in fields such as healthcare, finance, and 自律システム, where decision-making relies heavily on the reliability of probability estimates. A well-calibrated model helps stakeholders trust the predictions, leading to better decision-making and risk management.