La décroissance des paramètres, souvent appelée weight decay, is a regularization technique utilisé en apprentissage automatique and apprentissage profond models to prevent overfitting. It works by adding a penalty to the fonction de perte that is proportional to the square of the magnitude of the model parameters (weights). This penalty encourages the model to learn smaller weights, effectively leading to simpler models that generalize better to unseen data.
En termes pratiques, la décroissance des paramètres modifie le processus d'optimisation by reducing the values of the weights gradually over time. This is usually achieved by adjusting the weights according to their gradients, scaled by a small constant factor known as the taux de décroissance. The idea is to discourage the model from fitting noise in the training data, which can happen when the model has too much capacity (i.e., too many parameters) relative to the amount of training data available.
Mathématiquement, cela peut s'exprimer comme :
Loss = Original Loss + λ * ||W||²
where λ is the decay coefficient, and ||W||² denotes the L2 norm of the weights. The choice of the decay rate is crucial; if it’s too high, the model may underfit, while if it’s too low, it may still overfit.
Dans l'ensemble, la décroissance des paramètres est une technique largement utilisée dans diverses les applications d'IA, particularly in training neural networks, where it helps to maintain a balance between fitting the training data and ensuring that the model can perform well on new, unseen data.