Kaiming-Initialisierung

Kaiming-Initialisierung

Kaiming-Initialisierung, auch bekannt als He-Initialisierung, is a technique used to initialize the weights of neuronale Netze, particularly those employing Aktivierungsfunktionen like ReLU (Rectified Linear Unit). Proper Gewichtinitialisierung is crucial as it can significantly affect the convergence speed and overall performance of the network during training.

The method was introduced by Kaiming He and his collaborators in a 2015 paper. It aims to address the issues of vanishing and explodierenden Gradienten zu beheben that can occur during the training of deep networks. These issues arise when weights are initialized too small or too large, leading to ineffective learning.

Bei der Kaiming-Initialisierung werden die Gewichte aus einer Gaußsche Verteilung with a mean of 0 and a variance that is inversely proportional to the number of input units (fan-in) for each neuron. Specifically, the weights are initialized using the formula:

w ~ N(0, sqrt(2 / fan_in))

Where w represents the weight, N denotes a Normalverteilung, and fan_in is the number of input connections to the neuron. This approach helps maintain a balanced distribution of activations throughout the layers of the network, promoting effective gradient flow during backpropagation.

Kaiming Initialization is particularly effective in deep networks with ReLU and its variants but can be adapted for other activation functions with slight modifications. As a result, it has become a standard practice in training Deep Learning Modelle, was zu schnelleren und stabileren Konvergenzraten beiträgt.