マスクされた 言語モデル化 (MLM) is a technique used in 自然言語処理 (NLP) to train 言語モデルの by predicting missing words in a sentence. The core idea behind MLM is to randomly mask a portion of the input tokens (words or subwords) in a sequence and then train the model to predict the original tokens based on the surrounding context. This approach allows the model to learn deeper representations of language 単語間の関係とその文脈での使用を理解することによって訓練します。
MLM is a crucial component of transformer-based models, such as BERT (Bidirectional Encoder Representations from Transformers), which leverage this technique to achieve state-of-the-art performance on various NLP tasks, including text classification, 固有表現認識, and question answering. During training, a percentage of the input tokens are replaced with a special [MASK] token. The model then attempts to predict these masked tokens using the non-masked tokens in the sentence, thus learning to capture the underlying semantics and syntax of the language.
MLMの主な利点の一つは its ability to utilize bidirectional context, meaning the model can consider both the left and right context of a masked word. This contrasts with traditional unidirectional models that process text in a single direction. As a result, MLMs are able to generate more accurate and contextually relevant predictions, making them highly effective for various applications in AI and NLP.