A decadência de parâmetros, frequentemente chamada de weight decay, is a regularization technique usada em aprendizado de máquina and aprendizado profundo models to prevent overfitting. It works by adding a penalty to the função de perda that is proportional to the square of the magnitude of the model parameters (weights). This penalty encourages the model to learn smaller weights, effectively leading to simpler models that generalize better to unseen data.
Em termos práticos, a decadência de parâmetros modifica o processo de otimização by reducing the values of the weights gradually over time. This is usually achieved by adjusting the weights according to their gradients, scaled by a small constant factor known as the taxa de decadência. The idea is to discourage the model from fitting noise in the training data, which can happen when the model has too much capacity (i.e., too many parameters) relative to the amount of training data available.
Matematicamente, isso pode ser expresso como:
Loss = Original Loss + λ * ||W||²
where λ is the decay coefficient, and ||W||² denotes the L2 norm of the weights. The choice of the decay rate is crucial; if it’s too high, the model may underfit, while if it’s too low, it may still overfit.
No geral, a decadência de parâmetros é uma técnica amplamente utilizada em várias aplicações de IA, particularly in training neural networks, where it helps to maintain a balance between fitting the training data and ensuring that the model can perform well on new, unseen data.