Every time you search something, there’s a quiet tug-of-war going on behind the scenes between speed and actual understanding.
The Speed Problem
Databases were already great at being fast.
Index a column, look it up, done.
But free text broke that. You can’t index “meaning” the way you index a price or a date.
So engineers flipped the problem.
Instead of scanning every document when you search, they pre-built a map from every word to every document that contains it.
Search for "photosynthesis", and you’re not really searching anymore — you’re just jumping straight to the places that already have the answer.
That fixed the speed problem, but ranking was still messy.
From Counting Words to TF-IDF
Early search was basic: the word is either there or it isn’t.
Then came counting.
Mention a word more times and you rank higher.
Words that appear everywhere get discounted because they don’t mean much — "a", "the", "an" etc.
That approach is called TF-IDF (term frequency–inverse document frequency).
It was a big step up, but it still broke in obvious ways.
A page that repeats a word 200 times isn’t twice as relevant as one that repeats it 100 times.
And five mentions buried in a long essay don’t feel the same as five mentions in a short, focused post.
Raw counting couldn’t tell the difference.
Enter BM25
So the math evolved again. Frequency that levels off instead of climbing forever, and document length built right into the formula so long pages don’t automatically win. That’s BM25.
The formula looks like this:
TF × (k₁ + 1)
score = IDF × ─────────────────────────────
TF + k₁ × (1 - b + b × (docLen / avgDocLen))You don’t really need to memorize the formula. The important part is what it’s trying to fix.
Frequency levels off instead of climbing forever.
If a word appears once, that’s useful. If it appears five times, that’s more useful. But going from 100 mentions to 200 mentions shouldn’t suddenly make the document twice as relevant.
And document length matters.
Five mentions in a 100-word document don’t feel the same as five mentions in a 10,000-word document. BM25 takes that into account so long documents don’t automatically win just because they contain more words.
It’s still quietly powering the ranking inside Elasticsearch and Apache Solr today.
But Search Still Didn’t Understand Meaning
But here’s the limit:
None of this understands what a word actually means.
Search "car" and it will never show you a page that only says "automobile".
No synonyms, no sense of meaning — just very smart counting.
That’s exactly the gap vector search was built to close.
And that’s a whole different story.
References
https://arpitbhayani.me/blogs/bm25 https://arpitbhayani.me/blogs/idf https://youtu.be/iHHqnyThrqE?si=5pM9zavxFtB5hnCV