Rex Liu

Index less

BetterSpotlight, the file search I wrote for my Mac, and the night it failed every query I gave it.

I wrote a test suite for my search engine out of the files I actually go looking for, and it found none of them.

The queries came from my own folders, and forty-three of them imitate someone typing into the search box. Twelve were a file's exact name. The rest were phrases from inside documents, typos, file names mangled with junk punctuation and capitals, and some vague descriptions of what a file was about, the way you search when you remember the contents and not the name. A query passed if the file I meant came back near the top. The score was zero out of forty-three, and that included the twelve where the query was the file name.

Some context. BetterSpotlight is my replacement for Spotlight, for developers and other technical Mac users, and I started it in January in Swift. A month in I threw the Swift version away, most of the way to done, and started over in C++ with Qt, because I write C++ and I don't write Swift. In the decision record for the switch I ruled out Electron "categorically", on the grounds that "the team has strong objections to JavaScript/TypeScript." The team was me.

The C++ version came together in about three days. An indexer walks the disk, watches it for changes, and pulls the text out of documents, with OCR for images. A query service ranks matches with SQLite's FTS5 and its BM25 scoring (Robertson and Zaragoza's survey is the standard account of where BM25 comes from), and by the next evening I had embeddings on top for the vague queries. The two run as separate processes.

The ranking was the obvious suspect, since that's where all the interesting code was. The problem was what I'd told it to read. With nothing configured, the indexer's root was my whole home directory. On a machine I write software on, that's every repository, every dependency folder, every cache and config file I own, with a few thousand documents somewhere in the middle. A search for a document had to beat every other file that shared its words, and on my disk most of those files were source code, build output and caches.

So I changed what it reads. The default roots became Documents, Desktop and Downloads. Hidden directories stopped being indexed unless you add one as a root on purpose. And the query service started looking at the query itself: anything that reads like a sentence searches the document folders, and anything that looks like a path or a piece of code still searches everything that's indexed. Then I wiped the index, re-indexed the three folders and rebuilt the embeddings for the two thousand or so files left.

Same forty-three queries: thirty-eight passed. All twelve exact names, all eight phrases, all five of the mangled names. Eight of the ten typos, five of the eight vague descriptions. The 95th-percentile latency went from 260 ms to 144, which makes sense when there's less to search.

That night I wrote it all up as a memo, addressed with a straight face to a "Senior Engineering Executive". In it I put the remaining misses down to typo edge cases, vague queries, and iCloud files that existed only as placeholders with nothing on disk to read, and I recommended holding off on a cross-encoder reranker (a second model that rescores the top few results) until the cheaper fixes had been measured again. I added the cross-encoder the next afternoon.

The other thing I found that day is better. The same evening I wrote the test, I'd also added per-column weights, so that a match in a file's name could count for more than a match deep in its contents. FTS5 stores those through a special insert (the docs call it the rank configuration option), and I'd written the command wrong:

-- what I wrote
INSERT INTO search_index(search_index, rank) VALUES('fts5', 'bm25(%1, %2, %3)');

-- what FTS5 wants
INSERT INTO search_index(search_index, rank) VALUES('rank', 'bm25(%1, %2, %3)');

Searches sort by that stored rank, so the weights had never applied, and the thirty-eight passes came without them. Everything I'd changed between the failing run and the passing one was about which files it searched.

Out of the box, BetterSpotlight now reads three folders. If you want your code in it, you add the folder yourself.