Measured, with the conditions.

Retrieval quality is measured on public collections we did not build — the BEIR test sets — using the whole test split of each corpus, every query that has relevance judgements, and those judgements exactly as published. The comparison point is the published Lucene BM25 score for the same split, quoted as it stands, rather than a baseline we built and tuned ourselves. No query is dropped, no favourable subset of a corpus is selected, queries that return nothing are averaged in as zeros instead of being excluded, and errors are counted separately so a crash cannot be mistaken for an empty result. Where a figure would not survive being run again, we say that instead of publishing the figure. Where a condition was not recorded at the time of the run, we leave it out rather than reconstruct it.

This is the only environment the product has been verified on. macOS, Linux and ARM are untested: the POSIX build script exists and is checked for the symbols it must export, but that check runs on Windows, no measurement on this page was taken on those platforms, and none is claimed for them. Every number here comes from the one machine described above, which for part of the runs was carrying unrelated background load — noted per figure where it matters.

The table is short on purpose. The exact CPU model, the compiler flags and a build identifier were not recorded alongside these runs, so they are not stated here rather than reconstructed after the fact. That is a real gap for anyone trying to match the milliseconds: record your own hardware when you reproduce. It should not affect the quality figures, which were identical across two runs on the same machine under different load.

Conditions for both rows. All test-split queries with judgements, top_k=50, embeddings off, one document stored per corpus entry, the same code path for both corpora, Windows x64, Python 3.14. Errors were counted apart from empty results: zero errors on both corpora. The reference is the published Lucene BM25 score for the same split (k=0.9, b=0.4); we quote it, we did not re-run it here.

What the ratio says. BM25 in a mature implementation is the standard lexical baseline, and at 89% and 84% of it this engine is behind that baseline, not ahead of it. We publish the ratio because it is the number a reader needs in order to decide whether the rest of the system is worth looking at. It is not a win.

The 22 empty results are real. On NFCorpus, 22 of 323 queries return nothing: their terms do not occur in the corpus at all. Each counts as a zero in the average above rather than being dropped, which is why the column is in the table and not in a footnote.

Repeatability. Two independent runs on the same day, 2026-08-27, produced identical quality numbers — every digit. An earlier run over the same two corpora on 2026-08-26, driven through the shipped search tool rather than the comparison harness, under the same conditions otherwise — whole test splits, all 300 and 323 queries, top_k=50, embeddings off — scored 0.577 and 0.276; the same 89% and 84% of the reference after rounding.

Averages hide documents. nDCG@10 is a mean over queries, and two systems with the same mean can return different documents. Alongside the mean we compare the returned lists, result by result, against a run frozen from the previous storage implementation before it was replaced: on the same corpora, queries and settings as the table above, measured 2026-08-27, the top-10 lists agree 83.9% on SciFact and 87.8% on NFCorpus. The comparison also runs over ranks 11–50 and over the full 50, because a whole band of lower-ranked results can disappear without moving the mean much; those two agreement figures are not published here, so the 83.9% and 87.8% describe the top 10 only.

Conditions. One query at a time through the same search call an application would make, top_k=50, embeddings off, corpus indexed in one batch first, median over all timed queries; environment as in section 02. Throughout these runs the machine was carrying roughly 60% CPU from unrelated work, so the milliseconds are an upper bound rather than a best case.

Why there is no single latency number here.

The same procedure over the same corpus, run twice against identical code — BEIR SciFact, top_k=50, embeddings off, measured 2026-08-27 with the machine carrying about 44% CPU from unrelated work — has produced medians of 22.8 ms and 31.3 ms: a 37% spread with nothing changed between the runs. On this machine, under this kind of background load, a single median is not precise enough to carry a claim about any change smaller than that spread. The quality figures repeat digit for digit; the timings do not. Read the milliseconds above as an order of magnitude, and treat any latency comparison that rests on one median — ours or anyone else's — as unfinished until it shows a spread.

Breakdown of a search.

Conditions. BEIR SciFact, 5,183 documents held as 12,925 chunks, 200–300 real queries, top_k=50, embeddings off, measured 2026-08-28. Timings come from a per-stage instrument inside the search call, and the stages sum to 98.5% of wall clock; the remainder is the unaccounted row. The machine was carrying unrelated load, so the proportions are the point, not the absolute milliseconds.

Work done against work required.

Proportions say where time is spent but not whether it had to be spent, so the same run counts how much of the work is logically necessary. Counts are taken from the index directly, with no cap applied above it: BEIR SciFact, 5,183 documents as 12,925 chunks, 200–300 real queries, top_k=50, embeddings off, measured 2026-08-28 — the same run as the stage breakdown above.

  one query, top_k=50, BEIR SciFact, embeddings off, measured 2026-08-28

  12,925 chunks in the index
       |
       +--> 8,892 scored ............ 69% of the index
       |        |
       |        '--> 99.3% of them score below the cut
       |             worst query 63.2% — top score is 2.68x the cut
       |
       +--> 128 ids resolved ........ true need 60 (median), 78 (99th pct)
       |
       '--> 50 documents returned

How not to read the 69%. It is tempting to read it as work that could simply be deleted. Most of it cannot. A query is an OR over its terms, and on this corpus a query becomes 75 distinct character trigrams at the median — 130 at the 95th percentile, 166 at most, over the same SciFact run measured 2026-08-28. An OR over that many trigrams reaches essentially the whole corpus by nature: over 150 real SciFact queries on those 5,183 documents, 100.0% of documents are touched for trigrams and 99.9% for words. No change of index structure moves that. What is structurally removable is not how many documents are touched but how much each one costs. The 99.3%-below-the-cut figure is a ceiling on what an early-termination scheme could skip, not a promise that any scheme reaches it, and such schemes are known to weaken as the number of query terms grows — which is exactly this index's shape. Whether the bounds would tighten enough on this corpus is not answered by this measurement.

Resolving ids is already close to tight — 128 fetched against a true need of 60 at the median. That stage is 20% of the time and has roughly a factor of two in it, so it is not where an improvement would come from.

What a word index would change (simulation, not a built index).

The postings a query walks were counted for the current character trigrams and for words, over 150 real SciFact queries on the same 5,183 documents, measured 2026-08-28. This is a simulation. Term frequencies were computed from the corpus using the same normalisation the index applies (NFKC, then lowercase, n=3), with words split on [a-z0-9]+ and no stemming. It answers what the postings would look like, not what the product does today. Nothing here is a latency prediction.

The candidate set does not shrink; the same documents are visited, each more cheaply. Anyone reading "12.7x" as "a twelfth of the work" will be disappointed. The reason to state it as an addition rather than a replacement is that a word path can serve English queries while character trigrams keep serving Japanese and substring queries, which is how several analysers over one field are normally run elsewhere. Today the product ships the trigram index only.

This is not an official BEIR submission. The numbers were produced by our own harness, on our own machine, and have not been submitted to or checked by the BEIR maintainers or any third party. The reference column quotes published Lucene BM25 scores; we did not run Lucene here, so the two sides of the comparison were produced by different people on different hardware at different times. A reader who needs the comparison to be airtight should re-run both sides themselves.

Two corpora, both small, both English, both scientific. SciFact has 5,183 documents and NFCorpus 3,633. BEIR contains many more collections, and systems that look good on some of them look poor on others. Our own two disagree by five points of the ratio (89% and 84%) with nothing changed but the corpus, which is the clearest available evidence that the ratio is a property of the corpus as much as of the engine. Nothing here predicts a third corpus, and in particular nothing here predicts your documents.

Lexical only. Embeddings are off in this version, so this compares a keyword path against a keyword baseline. It says nothing about dense retrieval, hybrid retrieval, or reranking, all of which usually score above BM25 on these same collections. If your comparison set is a modern embedding pipeline, this page does not contain the number you want.

No Japanese measurement. The index is built from character trigrams specifically so that Japanese and substring queries work without word segmentation. Both benchmark corpora are English, so that property is not exercised anywhere on this page, and there is no published benchmark for it yet.

One machine, and only the parts of it we wrote down. Every figure comes from a single CPU under a desktop operating system, part of the time under unrelated load. There is no second machine, no server-class run, no repetition across hardware, and no cold-cache versus warm-cache separation. The CPU model and build flags were not recorded with the runs, so section 02 cannot name them. The 37% spread between two runs of identical code is the honest size of the resulting uncertainty in the timing figures.

Small data. Both corpora are thousands of documents, not millions, and index in seconds at the rates above. Nothing on this page describes behaviour at scale, and the stage breakdown in particular is a profile of this corpus at this size — proportions between stages will move as a corpus grows.

One metric, at one depth. nDCG@10 over top_k=50 is what we report. Recall at greater depth, precision at 1, and latency under concurrency are not measured here. Nor is anything measured under concurrent writers: all runs are one process, one query at a time.

This is a reproduction protocol, not yet a reproducible result. The datasets, the metric and the settings below are public, and the steps are what our harness does in the order it does them — so the procedure can be inspected and run against any implementation. What cannot be done today is reproducing these numbers: the implementation being measured is not published yet, so an independent party cannot run it. Independent reproduction becomes possible when the repository is released.

1. Take the datasets unmodified. BEIR SciFact and BEIR NFCorpus from their public distribution: the corpus, the queries, and the test-split relevance judgements. Use the test split of each, all of it.

2. Check you have the corpus you think you have. Hash the sorted id-and-text pairs before comparing anything against anything. A score computed over a subtly different corpus is not comparable, and that failure is silent.

3. Build each document's text as the title followed by the body, with the title omitted when empty — one stored document per corpus entry. Corpus ids that are not valid document ids are rewritten to a safe character set and mapped back before scoring, so no document quietly disappears in the round trip.

4. Settings: embeddings off, top_k = 50, everything else at its default. Index the whole corpus in one batch and record the documents per second and the number of chunks — SciFact should come out at 12,925 chunks.

5. Run every query that has judgements — 300 for SciFact, 323 for NFCorpus — timing each call. Keep empty results as empty. Count exceptions separately from empty results, so a failure can never be averaged in as if it were a query with no matches.

6. Score nDCG@10 with the standard formula against the published judgements, ideal DCG taken from the graded judgements of that query, averaged over all queries including the ones that returned nothing.

7. Compare with the published Lucene BM25 scores for the same splits: 0.665 on SciFact and 0.325 on NFCorpus — BEIR paper Table 2, Anserini BM25 (k=0.9, b=0.4).

8. Record your machine, since we could not fully record ours. Operating system, CPU, Python version, compiler and flags, and what else the machine was doing. The quality numbers should not depend on any of it; the milliseconds will.

9. Expect the quality numbers to reproduce and the timings not to. Two runs should agree on nDCG to every digit. If they do not, the corpus, the split, or the settings differ. Medians will move by tens of percent between runs on one machine; run enough of them to see the spread before drawing any conclusion from a difference.

Comparing two versions of a system rather than a system against a baseline: also compare the returned lists, not only the mean. Check agreement at ranks 11–50 as well as the top 10 — a change that drops results from the middle of the list can leave the mean almost where it was.