Les taux d'apprentissage cycliques (CLR) sont une technique d'entraînement dans apprentissage profond that adjusts the taux d'apprentissage dynamically during the training process. Instead of using a taux d'apprentissage constant, CLR allows the learning rate to fluctuate between a lower and upper bound, creating a cycle that can améliorer l'efficacité de l'entraînement du modèle et de la performance.
The concept is based on the idea that periodically increasing the learning rate can help the processus d'optimisation escape local minima, while decreasing it can facilitate convergence. Typically, during each cycle, the learning rate starts at a minimum value, increases to a maximum value, and then returns to the minimum. This cyclical pattern can be repeated multiple times throughout the training process.
Le CLR présente plusieurs avantages, notamment :
- Convergence plus rapide : By allowing the learning rate to vary, models can converge more quickly than with a fixed learning rate.
- Performance améliorée : The technique can help models achieve better generalization en évitant le surapprentissage.
- Réduction du besoin d'ajustement : Since CLR adapts the learning rate during training, it can reduce the need for extensive réglage des hyperparamètres.
Common implementations of CLR include triangular cycles, where the learning rate increases and decreases in a triangular waveform, and exponential cycles, where the learning rate follows an décroissance exponentielle pattern at certain points. Overall, Cyclical Learning Rates represent a powerful method for optimizing the training of deep learning models, making them a popular choice among practitioners.