TF-IDF (Frecuencia de Término - Frecuencia de Documento Inversa)
TF-IDF is a statistical measure that assesses the importance of a word within a document relative to a set of documents, often referred to as a corpus. It is widely used in recuperación de información, procesamiento de lenguaje natural, and minería de textos.
La medida comprende dos componentes: Frecuencia de Término (TF) y Frecuencia de Documento Inversa (IDF). Term Frequency calculates how frequently a term appears in a specific document. The intuition is that the more times a word appears in a document, the more relevant it is to the content of that document. Mathematically, it is expressed as:
TF(t, d) = (Número de veces que aparece el término t en el documento d) / (Número total de términos en el documento d)
On the other hand, Inverse Document Frequency quantifies how much information a word provides, based on how common or rare it is across all documents. Words that are very common across many documents (like ‘the’ or ‘and’) are less informative. IDF is calculated as:
IDF(t, D) = log(Total de documentos en D / Número de documentos que contienen el término t)
Combinando estos dos componentes, la puntuación TF-IDF se calcula como:
TF-IDF(t, d, D) = TF(t, d) * IDF(t, D)
This score helps highlight keywords that are both relevant to a specific document and not overly common in the broader corpus, thus making it a powerful tool for text analysis, search engines, and sistemas de recomendación. For example, documents that contain a high TF-IDF score for a particular term are likely to be more relevant to queries involving that term.