Am I overreacting for being upset my girlfriend called me her ‘friend’? We've been together for 8 months. Yesterday, she introduced me to her coworker as ‘my friend.’ I just stood there smiling like an idiot. Later I told her it bothered me. Her response: It's just a word. Why are you making it such a big deal? She wasn't just dismissing me. She was treating our relationship like a low frequency term in a massive corpus, technically present but not important enough to stand out. To understand what actually matters in a document, we use TF-IDF. It measures how important a word is, not just by how often it appears but by how rare it is across all documents. TF-IDF is a numerical statistic that reflects how important a word is to a document relative to a collection of documents. Formula TF-IDF = TF(t, d) × IDF(t) TF(t, d) = c(t) / k(d) IDF(t) = ln(N / (1 + df(t))) Full Form TF-IDF = Term Frequency-Inverse Document Frequency Where: - c(t) = count of term t in doc d - k(d) = total terms in doc d - N = Total no. of docs - df(t) = No. of docs containing term t Let's take an example: A news platform wants to identify the most important keywords in an article about space exploration. Corpus (4 articles) 1. NASA launches Mars rover mission 2. SpaceX rocket launches satellite 3. NASA announces Moon base plans 4. Weather forecast predicts rain Step 1: Calculate TF for NASA in A1 - Count of NASA = 1 - Total terms = 5 - TF = c(t) / k(d) - TF = 1 / 5 - TF = 0.20 Step 2: Calculate IDF for NASA - N = 4 documents - ‘NASA’ appears in Article 1, 3 - df(NASA) = 2 - IDF = ln(N / (1 + df(t))) - IDF = ln(4 / (1 + 2)) - IDF = ln(4 / 3) - IDF = 0.29 Step 3: Calculate TF-IDF for ‘NASA’ - TF-IDF = TF(t, d) × IDF(t) - TF-IDF = 0.20 × 0.29 - TF-IDF = 0.06 Step 4: Calculate TF-IDF for ‘launches’ - TF in Article 1 = 1/5 = 0.20 - ‘launches’ appears in Article 1, 2 - df(launches) = 2 - IDF = ln(N / (1 + df(t))) - IDF = ln(4 / (1 + 2)) - IDF = ln(4 / 3) - IDF = 0.29 - TF-IDF = TF(t, d) × IDF(t) - TF-IDF = 0.20 × 0.29 - TF-IDF = 0.06 Step 5: Calculate TF-IDF for ‘Mars’ - TF in Article 1 = 1/5 = 0.20 - ‘Mars’ appears in Article 1 - df(Mars) = 1 - IDF = ln(N / (1 + df(t))) - IDF = ln(4 / (1 + 1)) - IDF = ln(4 / 2) - IDF = 0.69 - TF-IDF = TF(t, d) × IDF(t) - TF-IDF = 0.20 × 0.69 - TF-IDF= 0.14 Final Answer “Mars” has the highest TF-IDF (~0.14) because it's unique to this article. “NASA” and “launches” score lower (~0.06) because they appear in multiple documents. The rarer the term, the more it defines that specific document. Congratulations 🎉, you've just learned TF-IDF. Bonus: Applications in AI/ML 1. Search Engines: Ranking documents by relevance, pages where your search term is frequent AND rare elsewhere rank higher. 2. Text Classification: Transforming raw text into weighted feature vectors for spam detection, sentiment analysis, and topic categorization. 3. Keyword Extraction: Identifying the most distinctive terms in a document for automatic tagging and summarization. 4. Recommendation Systems: Finding similar documents or content by comparing TF-IDF vectors using cosine similarity.