Dunkles Wissen (Destillation) ist eine fortgeschrittene Technik im Bereich von maschinellem Lernen, particularly in the context of Modellkomprimierung and Transferlernen. This process involves transferring knowledge from a large, complex model (often referred to as the ‘teacher’) to a smaller, simpler model (the ‘student’). The primary goal is to achieve a high level of performance in the student model while maintaining a reduced computational footprint.
The term ‘dark knowledge’ refers to the soft labels produced by the teacher model during training. Unlike hard labels, which correspond to the actual class of an input, soft labels provide a probability distribution over all possible classes. This additional information captures the nuances and relationships between classes that are not evident from hard labels alone. By training the student model on these soft labels, it can learn to mimic the teacher’s behavior and generalize better on unseen data.
In practice, the distillation process involves minimizing a loss function that measures the difference between the outputs of the teacher and the student models. This often includes a temperature parameter that controls the smoothness of the softmax output, allowing the student to learn from the softer probabilities. Dark Knowledge (Distillation) is particularly useful in scenarios where deploying large models is impractical, such as on mobile Geräte or embedded systems, thus enabling efficient inference without significantly sacrificing accuracy.