Search quality benchmark

Summary

• On 597 LitSearch literature-search questions, Recall@20 is 0.5095 across all questions and 0.6120 on the 491 questions whose gold papers are all in the index. 465 of the 574 gold papers are indexed.
• Against the LitSearch paper's retrievers on its closed corpus, our all-questions result is behind GritLM-7B on all four comparable cuts and behind BM25 on three of four. That setting differs from ours; see the notes beside the table.
• On three Cochrane reviews, frozen queries found 89%, 100% and 78% of the indexed included studies within 500 pooled records; over all included studies, including those not in the index, 73%, 82% and 50%.
• Most review misses are documents we store but had never embedded, so search could not return them; the repair is described under What we're fixing.

Finding a specific paper: LitSearch

LitSearch (Ajith et al., EMNLP 2024) has 597 expert-checked literature-search questions about machine learning and NLP papers, each with one or more gold papers. Some questions were generated from paragraphs with inline citations and then checked by experts; others were written by authors about their own recent papers. Each question is marked specific or broad. On October 9, 2026 we sent every question to the whole live index as an agent's default search (documents, 20 results, automatic ranking mode, cross-encoder reranking on) and recorded where the gold papers ranked.

R@k is the share of a question's gold papers in the top k results, averaged over questions. 465 of 574 gold papers are indexed. Search latency was 4.0 s at the median and 6.1 s at the 95th percentile.

Next to the numbers in the LitSearch paper

Where indexed gold papers are lost

186 questions had their gold paper indexed but outside the top 20. We reran them at depth 250 for each retrieval configuration. The competing results are on topic: the remaining loss is finding one specific paper among many similar ones. On these long natural-language questions the lexical channels add little; within 250, the binary channel reaches 79 of these papers, BM25 over full text 20 and the learned sparse channel 10.

Search-stage recall on three Cochrane reviews

For each review, the gold list is its included studies from the CLEF 2019 Technology Assisted Review Task 2 content-level qrels. Each review has 9–10 frozen queries, limited to the review's own search date. We checked which included studies are in the index by exact PubMed or DOI identifier, ran every query, interleaved the ranked results round-robin into one pool, and measured how many indexed studies appear within the first 100, 500, 1,000 and 2,000 pooled records, and how many records must be read to reach 80% and 95% of them. Run on October 5, 2026.

Over all included studies, including those not in the index, recall at 500 records is 73%, 82% and 50%. A dash means the share was not reached in the pool.

Misses

Every one of the 23 missed studies is in our document database; none is missing from the corpus. They were missed for these reasons, checked on October 5, 2026:

Frozen queries

Data

The review specifications contain the frozen queries, search dates and gold lists; the results contain recall at every depth, which query found which study, the depth each query reached, and every miss.

• CD001261.json · specification: queries, search date, gold list
• CD012768.json · specification: queries, search date, gold list
• CD011787.json · specification: queries, search date, gold list
• results.json · full output of the October 5 run
• gold-status.csv · every included study: retrieved, in index but not retrieved, or not in index
• LICENSE-CLEF-TAR.txt · MIT licence of the CLEF TAR qrels the gold lists come from
• litsearch-summary.json · LitSearch recall by group and set, October 9

Methodology and reproduction

Both runs used the internal route of the production search service with unbilled benchmark requests, not a separate test system. The public REST API accepts the same request fields; your requests are billed searches and are rate limited, so pace them and honour Retry-After on 429 responses. The index changes every day, so a rerun will not match these numbers exactly.

• LitSearch. Questions and gold papers come from the princeton-nlp/LitSearch query set. Gold papers are matched by DOI or arXiv identifier, looked up through Semantic Scholar; a paper counts as found when any of its identifiers appears in a result. Coverage is checked first with an exact identifier probe.
• Reviews. Each query is fetched to depth 500 in pages of 50, with filter_issued_before set to the review's search date. Studies are matched by pubmed:// or doi:// URI.

# LitSearch: one question as an agent's default search
curl -X POST https://api.machinelibrary.ai/v2/search/ \
  -H "X-Api-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "<LitSearch question>", "index_names": ["documents"], "limit": 20}'

# Reviews: one page of a frozen query, limited to the search date
# (CD001261, 2018-02-14 = 1518566400); repeat with offset 50, 100, ... to 450
curl -X POST https://api.machinelibrary.ai/v2/search/ \
  -H "X-Api-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "typhoid vaccine randomized controlled trial", "index_names": ["documents"], "limit": 50, "offset": 0, "filter_issued_before": 1518566400}'

# Coverage: is a gold study indexed? Count only an exact URI match.
curl -X POST https://api.machinelibrary.ai/v2/search/ \
  -H "X-Api-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "", "index_names": ["documents"], "filter_uri_prefixes": ["pubmed://10506405"], "limit": 10, "cross_rerank": false, "deduplicate": false}'

Limitations

• Coverage gaps: 465 of 574 LitSearch gold papers and 18 of 22, 18 of 22 and 18 of 28 included studies are indexed. Recall over all gold is reported beside recall over indexed gold.
• Title-only records are never embedded or indexed, because the index requires a body. On October 9, 52,123 of the 54,122 ACL DOI records were title-only Crossref records; in a block sample of journal articles, 1,152 of 2,823 (41%) were title-only.
• Depth cap: the search engine refused pages at offset 250 or more with 429, so most review queries stopped at 250 records. Recall at 1,000 and 2,000 measures the pooled records, not the depth of each query. Only the first 256 hits are reranked; deeper pages keep retrieval order.
• Search dates use each review strategy’s MEDLINE entry-date limit. CD011787 had none and is capped at its publication date.
• Three reviews are not a benchmark. Recall depends on the query set, and this measures discovery, not screening: a ranked result is not an inclusion decision.
• LitSearch covers machine learning and NLP only. Our index holds newer work on every topic; a run restricted to papers issued before 2024-01-01 raised R@5 by 2–6 points on the ACL and non-ACL sets but lowered the ICLR set, whose gold papers are from ICLR 2024.
• The LitSearch questions and gold lists are not redistributed here; get them from the dataset. Its numbers are not directly comparable with the paper’s closed-corpus setting.

What we're fixing

Documents stored but never embedded

17 of the 23 review misses have text but no vectors, so they never reached the search index. In a block sample of journal articles, 142 of 2,823 (5.0%) are in the same state; in an earlier sample, 118 of 121 such documents came from the January–February 2026 bulk load. The repair is a backfill of unembedded documents in the embeddings worker.

Title-only ACL records gaining abstracts

A new ACL Anthology source adds Anthology abstracts to title-only ACL records. Its first 400 collection files merged abstracts into 3,628 records and added 3,918 papers, and a merged record was searchable minutes later. ACL preprints and published versions remain separate records.

Under investigation: fusion of retrieval channels

On 31 LitSearch questions, one retrieval channel found the indexed gold paper and fusion lost it. Binary-only retrieval wins 50 questions and loses 72 against the default hybrid, so the channels are complementary; merging a binary candidate pool into the cross-encoder pool is the current lead.