What Is TF-IDF?
Written byFlavio Amiel Founder, Roborank
TF-IDF (term frequency–inverse document frequency) is a numerical statistic from information retrieval that weighs how important a word is to one document within a larger collection. It multiplies how often a term appears in a document by how rare that term is across the whole corpus, so distinctive words score high and common words score low.
- The score combines two parts: term frequency (how often a word appears in a document) times inverse document frequency (a log-scaled measure of how rare the word is across the corpus).
- Inverse document frequency was introduced by Karen Spärck Jones in a 1972 paper, decades before the modern web.
- Google’s John Mueller has called TF-IDF an old metric and cautioned against optimizing for it, noting it is better suited to spotting stop words.
- You cannot compute true IDF yourself, because it requires knowing every document in Google’s index.
- Third-party ‘TF-IDF content tools’ approximate the score against a small sample of ranking pages, not Google’s real corpus.
How TF-IDF Works
TF-IDF answers a narrow question: within a collection of documents, which words best characterize any single one of them? It does this by balancing two opposing forces. Term frequency rewards a word for appearing often in a document — a page that says “espresso” ten times is probably about espresso. Inverse document frequency then penalizes words that appear in lots of documents — “the” and “coffee” show up everywhere, so their weight is discounted, while a rarer, more discriminating term like “portafilter” keeps its punch. Multiply the two and you get a score that is high for words that are both prominent here and unusual elsewhere.
That balance is what made TF-IDF durable. It is a cornerstone of classic information retrieval and still powers plenty of search and text-mining systems. But it is a general statistic, not a window into Google’s ranking. Google’s own John Mueller has described TF-IDF as a fairly old metric that predates how much search has changed, and pointed out that its most defensible modern use is mundane — helping identify stop words to discard, not scoring pages to rank.
The TF-IDF Formula
In its common form:
TF-IDF(term, document) = TF(term, document) × IDF(term)
where IDF(term) = log( total documents ÷ documents containing the term ).
The logarithm is what keeps the rarity bonus from exploding: a term in 1 of 10,000 documents scores higher than one in 1,000, but not a thousand times higher. The catch for SEO is buried in that “total documents” figure — the corpus is Google’s entire index, which no outside tool can see. Every commercial “TF-IDF score” is therefore an estimate against a handful of ranking pages, not the real value.
Example of TF-IDF
The concept traces to a genuinely foundational document: Karen Spärck Jones’s 1972 paper, “A statistical interpretation of term specificity and its application in retrieval,” in the Journal of Documentation. Spärck Jones argued that a term’s value for retrieval should depend on its specificity — how narrowly it is distributed across a collection — which is exactly the inverse-document-frequency idea that still carries her name.
A short illustrative calculation shows the mechanism (numbers chosen for clarity, not from a real index). Suppose a 200-word article uses “portafilter” 4 times, so its term frequency is 4/200 = 0.02. Suppose the word appears in 100 of a 1,000,000-document corpus, giving IDF = log(1,000,000 ÷ 100) = log(10,000) = 4. The TF-IDF weight is 0.02 × 4 = 0.08. Swap in a common word like “coffee” that appears in 500,000 of the million documents, and IDF = log(2) ≈ 0.3 — a far smaller weight even at the same frequency. The distinctive term wins, which is the whole point. What the example also makes concrete is the limitation Mueller flags: change the corpus and every score shifts, so a number tuned against ten competitor pages tells you nothing reliable about how Google, working from billions of documents, actually weighs your text.
I treat TF-IDF tools the way I treat a thesaurus: occasionally useful for ideas, never a checklist. The failure mode I see is a writer running a page through a TF-IDF optimizer, getting a list of “under-used terms,” and then bolting those words into sentences that did not need them — which is keyword stuffing dressed up in a spreadsheet. The genuine signal a TF-IDF tool surfaces is topic coverage: if every strong competitor mentions “warranty” and “return policy” and your page never does, that is worth knowing. But the number itself is not a target, and hitting some suggested weight buys you nothing with Google. Use it to find gaps in what you cover, then close them with real information, not by sprinkling terms to move a score.
Frequently Asked Questions
Does Google use TF-IDF?
Who invented TF-IDF?
What is TF-IDF used for?
Should I optimize my content for TF-IDF?
The Bottom Line
TF-IDF is a classic recipe for deciding which words make a document distinctive: reward terms that appear often here but rarely everywhere else. It underpins decades of search technology, yet Google does not rank pages on it directly, and you cannot even calculate a true version without Google’s full index. Mine it for coverage gaps, then ignore the number.
Sources
- A statistical interpretation of term specificity and its application in retrieval (Karen Spärck Jones, 1972) — Journal of Documentation
- TF-IDF: Is It A Google Ranking Factor? — Search Engine Journal
Rank & Cash — the weekly SEO breakdown
One practical teardown a week on ranking in search and getting cited by AI. No fluff.