Machine Library research corpus

Access, provenance & rights

Search access and bulk data access are separate products. You can inspect records through the web app, REST API, or MCP server before discussing a licensed bulk delivery.

Search access

Results include canonical source URIs and available bibliographic metadata. Full text is returned only where it is present in the corpus.

Bulk access

Scoped by source segment, intended use, and volume. There is no anonymous whole-corpus download.

Provenance

DOI, PubMed, ISBN, patent, and source-page identifiers are retained where available so records can be traced back to their source.

Rights

Availability and permitted use vary by source and content segment. A Machine Library delivery does not replace third-party rights or attribution requirements.

Scholarly & reference records

The core of the corpus: 204M scholarly and reference records with metadata, abstracts, and full text where available. Type estimates use a 23,386-record block sample. YouTube transcripts are document records and are outside this scholarly breakdown.

Stored text and abstracts

Presence of a non-empty text field in a 0.001% block sample of document records; denominators are the sampled records of each type. This does not establish that the original document has been parsed completely. Small segments have wider sampling uncertainty.

• Journal articles with abstract
  51% · 846/1666 sampled
• Patents with stored text
  100% · 35/35 sampled
• Encyclopedia articles with stored text
  100% · 90/90 sampled
• Books with stored text
  58% · 34/59 sampled
• Journal articles with stored text
  31% · 518/1666 sampled

Linked identifiers

Records retain available DOI, PubMed, ISBN, arXiv, patent, and source-page identifiers. Coverage varies by source.

Community discussions

Public conversation streams complement the scholarly record with current, informal knowledge.

Message and submission estimates use a 23,680-record block sample. Messages include public Telegram and Discord content. Comments are stored separately and are not added to the headline record total. Both catalog estimates are lower than in the September 12 snapshot (2.70B community records, 24.14B comments); they describe the current tables.

Text volume: July 2026 estimate

Estimated token counts by segment, assuming ≈4 characters per token. The July snapshot totalled roughly 7.3 trillion characters, or ≈1.8 trillion tokens. These historical estimates have not been remeasured and are not a count of tokens available for licensing.

Languages: August 2026 sample

100+ languages appear in language-tagged documents. This distribution uses only scholarly and reference records; Telegram, Reddit, Discord, and YouTube records are excluded.

• English
  84.2%
• German
  3.8%
• French
  2.2%
• Russian
  1.8%
• Spanish
  1.4%
• Portuguese
  1.1%
• Italian
  0.5%
• Chinese
  0.5%
• Japanese
  0.4%
• Other (90+)
  4.1%

Publication dates: July 2026 sample

Scholarly coverage is deep — a third of works were published in the 2020s, yet the corpus reaches back past 1900. Community content concentrates in recent years: 63% was published within the last five years.

Embeddings & storage

The database holds approximately 8.28 billion embedding rows across document and community records. Multiple representations can refer to the same passage, so this is not a count of unique passages. Storage figures below are measured database footprints, including indexes.

Original source files: August 8, 2026 inventory

The last measured object-storage inventory, separate from the search database, contained 76.9 million files totalling 128.9 TB. PDFs dominate with 71.9 million files and 119.7 TB, alongside 2.2 million EPUBs, 171,000 DJVU scans, and publisher HTML and JATS XML sources.

Methodology

The October 14 snapshot sums planner row estimates over all 32 leaf partitions of each record table. These are estimates, not exact counts. Type shares use reproducible block samples: 0.01% of document blocks and 0.001% of community blocks. Text-field coverage uses a separate 0.001% document sample. Samples may over- or under-represent rare sources. Database sizes are measured directly and use decimal TB. Token, language, publication-date, and object-storage measurements keep their original dates above; they have not been refreshed in this snapshot.