Unité Linéaire Rectifiée Leaky (Leaky ReLU) est un fonction d'activation commonly used in réseaux neuronaux, particularly in apprentissage profond. It is designed to address the problem of dying neurons, which can occur in standard ReLU (Rectified Linear Unit) functions where neurons become inactive and stop learning. The Leaky ReLU function allows a small, non-zero gradient when the input is negative, preventing neurons from becoming completely inactive.
La formule mathématique de Leaky ReLU est définie comme :
f(x) = x if x > 0, else α * x
Here, α (alpha) is a small constant (typically 0.01) that determines the slope of the function for negative inputs. This means that for negative values, instead of being zero as in standard ReLU, the output will be a small, negative value, thus maintaining some level of activation even for non-positive inputs.
One of the advantages of Leaky ReLU is that it helps to mitigate the vanishing gradient problem, making it easier for models to learn and generalize. This property can lead to better performance in deep networks compared to traditional fonctions d'activation. However, the choice of alpha can affect the training dynamics, and it’s often recommended to experiment with this parameter during model tuning.
In summary, Leaky ReLU is a simple yet effective activation function that enhances the learning capacity of neural networks by allowing a small gradient for negative inputs, thereby keeping neurons active and contributing to the modèle global performance.