This page lists the studies this project has run, with the conditions each number was measured under and a statement of what each study does not establish. Identifiers are permanent: when a later study changes a result, the row is not deleted or rewritten — its status becomes Superseded by HC-R0xx, naming the study that replaced it, so a number that was once quoted can still be found and dated.

Status takes one of four values. Published — measured, with conditions stated. Reproduced — additionally recomputed from the raw records by an independently written implementation. In progress — running; no numbers are reported until it finishes. Superseded — replaced by a later study, and kept for reference.

The date is the date of the most recent measurement in that study, not the date this page was edited. The underlying records are published machine-readably at facts.json.

Status: Reproduced. Runs measured 2026-08-29; paired statistics and the attribution analysis added 2026-08-30. Model: qwen2.5:0.5b [6].

A small language model answers factual questions better when the irrelevant context is removed before inference than when the entire corpus is placed in its context window — even though the pasted window strictly contains everything the retrieved window contains. If this holds, the advantage should not shrink as the corpus grows, because the pasted input grows with the corpus while the retrieved input does not. That retrieval grounding helps generation at all is established elsewhere [4]; what is open, and what this study tests, is whether retrieval still helps when the alternative is not an empty context but the complete one.

The hypothesis is falsifiable in two ways that were fixed in advance: the pasted window could win on some questions in a paired comparison, or the retrieved input could grow with corpus size. Both were checked.

Corpus. 100 documents, each stating a small set of attributes for one fictitious company, generated from one template. The names occur nowhere outside this corpus, so an answer cannot have been memorised during pretraining. That is not assumed — the closed-book control measures it.

Questions. 100 questions, each asking for one attribute of one company, so that exactly one document holds the answer.

Conditions. Every question is asked once under each condition, with the same model and the same decoding settings. Pasted window: all documents concatenated into the prompt. Retrieved: the documents are stored in the cabinet and only search results enter the prompt; the query is formed from the question by a fixed deterministic rule — a subject prefix plus an attribute keyword from a fixed list mirroring the corpus schema, taking the question text as its only input — which is the role an AI client plays in ordinary use. Closed book: no documents, to measure leakage. Wrong document: one deliberately incorrect document, to measure whether the model answers from what it is handed or from its priors.

Scoring. A response counts as correct when it contains the expected value with no digit adjacent to that value at either boundary, so “11948” never matches “1948”. No model takes part in judging. The judge credits any response that contains the exact value, so a response listing several values scores correct, and it fails a correct answer written in a different surface form; both failure directions apply identically to every condition.

Preregistration. Three inequalities were fixed before the runs: when the corpus doubles, retrieved input may grow at most 1.25× and pasted input must grow at least 1.8×, and retrieval accuracy must stay within 10 percentage points of the pasted window. The registration was amended once, before any data was collected, and the amendment is kept with its reasons.

Scales. The whole protocol was run at 50 documents and at 100 documents.

                 +--> [ all 100 documents pasted ] --> model --> judge
   question -----+
                 +--> [ cabinet search, top 10   ] --> model --> judge

                 +--> [ no documents at all      ] --> model --> judge
   controls -----+
                 +--> [ one wrong document       ] --> model --> judge

Both rows are the same 100 questions, the same model at temperature 0, and the same deterministic judge, measured 2026-08-29 on Windows x64; the only difference is what enters the prompt and the context-window setting the condition requires. The input ratio at this corpus size is 20× and the accuracy gap is 52 percentage points. Both token figures are means over the 100 questions, as reported by the runtime.

The comparison is paired on identical questions, and it is one-sided. The counts below are the same 2026-08-29 runs, paired per question, with the pairing computed in the 2026-08-30 attribution analysis:

Exact binomial McNemar on the two discordant cells gives p = 4.4e-16. The pasted window does not win a single question that retrieval loses, so this is not a trade between quality and input size at this scale.

Three controls locate where the 88% comes from. All three are the same 100 questions, the same model at temperature 0, and the same deterministic judge; the first two ran with the 2026-08-29 runs and the third was computed 2026-08-30.

The hit rate was recomputed on 2026-08-30 by rebuilding the identical corpus into a fresh database and re-running the same deterministic query rule at top_k = 10, so 88% decomposes as a 100% retrieval hit rate times an 88% read-out rate: every miss is the model failing to read an answer that was in front of it, not search failing to find it.

Doubling the corpus, measured 2026-08-29 under the conditions above:

Both columns are the same model at temperature 0 and the same deterministic judge, measured 2026-08-29; the 50-document column is the first 50 documents and their 50 questions. No interval is shown for the 50-document retrieval cell because none was computed for it.

All three preregistered inequalities passed. The token flatness on the retrieval side is structural rather than a discovery — search returns at most top_k documents, so the measurement confirms a mechanism that was designed in. The empirical content sits on the accuracy side.

The result is best read as a statement about read-out, not about search. Search found the answer every time, at both corpus sizes, so the ceiling in this experiment is set by how well a 0.5B model reads what it is given. Adding the other 99 documents to that same prompt did not add information the model could use; it removed accuracy, and it did so on more than half the questions. Whatever the mechanism, at this scale the model was not able to ignore irrelevant context for free.

The decline on the pasted side from 46% to 36% points in the same direction as the established finding that models use long contexts unevenly and degrade as those contexts fill [5], and two alternative explanations were checked and do not account for it: on the identical first 50 questions the 100-document window also scores 36%, so it is not question composition, and questions whose document sits in the front half of the pasted corpus score 18 of 50 against 18 of 50 for the back half, so it is not position within the window. Still, the two pasted-window intervals overlap. We report the decline as a point estimate consistent in direction with that literature, not as a measured slope. What the preregistered criteria establish is the growth contrast, and that accuracy at 100 documents did not suffer from externalization.

For the design of the cabinet, the practical reading is narrow and useful: on this corpus, giving a small local model less text was not a concession to a small context window. It was the better of the two options on both axes at once.

The corpus is synthetic and single-template. One hundred generated records about fictitious companies, and questions that ask for one attribute of one entity. Nothing here speaks to multi-hop questions, aggregation across documents, summarisation, contradictory sources, or long documents whose answer is spread across passages.

Retrieval in this corpus is easier than in a real one. Every question contains a globally unique company name, so the right document cannot be crowded out of the top 10. The 100% hit rate is a property of the corpus at least as much as of the engine. A corpus with near-identical names, revisions, contradictions, or numerically divergent duplicates could push that hit rate down, and since accuracy here is the hit rate times a read-out rate, the 88% would fall with it. That regime is untested.

This result must not be generalised to all retrieval-augmented workloads. It measures one retrieval regime, one corpus shape, one question type, and one prompt format. It is evidence that externalizing context can beat pasting it; it is not evidence about where the crossover lies for any other workload, and it says nothing about corpora large enough for retrieval itself to become the limiting stage.

One model. qwen2.5:0.5b only — a single scale in a single family. A different family, a different instruction-tuning recipe, or a larger model may read a long context far better and shrink or reverse the gap.

One generation pass per cell. The model inference behind each cell ran once. What was reproduced independently — twice, and agreeing to every digit — is the judging and aggregation from the raw per-question records, not the generation itself. Temperature 0 makes that pass the intended sample, not a low-variance estimate of one.

Token counts come from the runtime. Both arms report the prompt token count the local runtime itself reports, not an independent tokenisation, so the counts carry that runtime's accounting conventions. The ratio between the arms is what the study rests on, not the absolute counts.

The arms differ in window setting as well as content. A prompt holding the whole corpus needs a larger context window than one holding a handful of search results, so the two arms cannot be run at one setting. Model weights, decoding, and judge are identical; the runtime configuration is not.

The judge is a containment judge. Of the 100 responses examined, 1 contains two values of the same attribute and 4 contain a value belonging to another document, so the credit a lenient judge could give is small but not zero. It also fails correct answers phrased in an unexpected form.

Windows x64 only. The engine underneath has been built and tested on that platform alone so far. macOS, Linux, and ARM are unverified.

A repetition of the identical protocol at the 7B scale is in progress, and no number from it appears on this page.

The steps below are the experiment, not a description of it. They need a local model runtime, a small corpus you generate yourself, and any store that returns ranked documents deterministically.

What should come out, if the corpus and the rule are built as described: a closed-book accuracy of 0, a retrieval arm far ahead of the pasted arm on the same questions, a retrieval token count that barely moves between 50 and 100 documents, and a pasted token count that roughly doubles. The exact percentages depend on your corpus and your model build; the direction and the growth ratios should not.

Both studies ask one question: how far does a deterministic lexical index in a single file get, on standard benchmarks, with no embedding model and no external service to operate.

Measured 2026-08-27 on Windows x64, Python 3.14, embeddings off, top_k = 50, every query in the split counted. Scoring is nDCG@10, the BEIR primary metric [1]; ranking is BM25 [2] with Anserini-compatible constants, k₁ = 0.9 and b = 0.4 [3], over character trigrams. In absolute terms the gaps are −0.085 and −0.052 nDCG@10.

Three things keep the comparison honest. Tokenising into character trigrams gives word-level BM25 a structural advantage on English text, so this is not a like-for-like implementation comparison. The reference values are published Lucene BM25 figures, not re-runs under this harness, so the ratios are approximate across evaluation toolchains [1, 3]. And the 22 NFCorpus queries that return nothing are counted as zero rather than excluded. Neither number claims parity with a tuned dense-retrieval stack; both state a fraction of a published lexical reference, reached with nothing to operate. Both figures are from the same platform as HC-R001 and carry the same Windows-only caveat.

In progress. The design is an ablation ladder over the quality-against-size trade-off of the index: fixed-threshold pruning, similarity de-duplication, congestion-game pruning, and percolation-selected similarity thresholds, each measured separately and in combination, against a random-deletion control at matched document counts. The control is the point of the design — a pruning rule has to beat deleting the same number of entries at random before its mechanism is worth discussing.

No numbers are reported while it runs. The row moves to Published when the runs finish and an independent recomputation from the raw records agrees with the aggregation.

[1] N. Thakur, N. Reimers, A. Rücklé, A. Srivastava, I. Gurevych. “BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models.” NeurIPS Datasets and Benchmarks, 2021.

[2] S. Robertson, H. Zaragoza. “The Probabilistic Relevance Framework: BM25 and Beyond.” Foundations and Trends in Information Retrieval, 2009.

[3] P. Yang, H. Fang, J. Lin. “Anserini: Enabling the Use of Lucene for Information Retrieval Research.” SIGIR, 2017. The BM25 reference values in HC-R002 and HC-R003 — 0.665 for SciFact and 0.325 for NFCorpus — are the published Lucene BM25 figures.

[4] P. Lewis et al. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” NeurIPS, 2020.

[5] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, P. Liang. “Lost in the Middle: How Language Models Use Long Contexts.” TACL, 2024.

[6] Qwen Team. “Qwen2.5 Technical Report.” arXiv:2412.15115, 2024.