---
title: Benchmark results
---

This page lists only numbers that a committed file reproduces. Each result is one of three
kinds:

- a **measurement** of Loom, under a real provider unless its section says otherwise,
  recorded in a committed artifact;
- a **reference computation** the code produces on fixed input;
- a **scorer self-check** that proves a harness counts correctly.

Only measurements say anything about Loom's retrieval.

The [coverage matrix](/benchmarks/coverage) separates product surfaces with a published
measurement from those with only a harness or no public evidence yet.

## Measurements

Each result reports one metric. **Hit@k** asks whether at least one gold item appeared within
the first `k`; **recall@k** averages the fraction of all labelled gold items retrieved;
**all-gold@k** requires every gold item. F1 balances precision and recall, and MRR rewards
placing the first gold item near the top. Retrieval metrics and answer accuracy, or answer
scores from different readers or judges, are not comparable. Latency results describe one
server and ClickHouse deployment, not a capacity promise.

Each measurement names its artifact, the instrument it was recorded under and a
reproduction command. New committed results go in this section, one subsection each.

A benchmark name identifies the source corpus, not automatically the official task or
leaderboard protocol. Every derived task below says what Loom actually received and what
the scorer counted. “All” means the complete published split; “sample” or “slice” means
the stated selection. Answer scores are comparable only when reader and judge are held
constant; retrieval scores are not answer accuracy or issue-resolution rates.

### Published benchmark protocol index

Use this table to establish exactly what each published number measures before comparing
results. “Questions” are independent prompts scored against ground truth; “contexts” are
the conversations, documents or repository revisions searched to answer them. Follow the
benchmark link for models, category results, limitations, artifact paths and the exact
reproduction command.

| Benchmark                                                                             | Exact task                                                                                                                                                                                                                 | Corpus and evaluation size                                                                                                                                                                                                                                         | Loom path and search depth                                                                                                                                                                                                                                                                                     | Success definition or comparison                                                                                                                                                                                                                                                                                                                          |
| ------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [LoCoMo conversational QA](#locomo-conversational-qa)                                 | Answer questions about facts, events, relationships and time across long conversations; abstain on adversarial questions.                                                                                                  | All 10 conversations; 1,986 questions: 1,540 answerable and 446 adversarial.                                                                                                                                                                                       | Conversations ingested with `memory.set_from_messages`; `memory.ask`, or `memory.search` top 10 followed by fixed `gpt-4.1-mini`.                                                                                                                                                                              | `gpt-4.1`-judged answer accuracy for categories 1–4; correct abstention for category 5, which published LoCoMo comparisons normally exclude.                                                                                                                                                                                                              |
| [LongMemEval-S](#longmemeval-s)                                                       | Answer questions requiring information distributed across many earlier sessions and retrieve the supporting evidence.                                                                                                      | All 500 questions; about 47 sessions in each question's history.                                                                                                                                                                                                   | Histories ingested with `memory.set_from_messages`; `memory.ask`, or `memory.search` top 200 followed by `gpt-5`; supporting-session recall scored at 30.                                                                                                                                                      | Official LongMemEval judge; `memory.ask` is Loom's product flow, not a standard leaderboard configuration; the fixed-reader arm is reported separately.                                                                                                                                                                                                   |
| [MemBench](#membench)                                                                 | Answer multiple-choice questions about events the agent observed or participated in.                                                                                                                                       | Two scenarios; 140 questions per scenario, covering seven question types.                                                                                                                                                                                          | Scenario memories ingested, then answered with `memory.ask`.                                                                                                                                                                                                                                                   | Exact multiple-choice accuracy; no LLM judge or external baseline.                                                                                                                                                                                                                                                                                        |
| [MemoryAgentBench samples](#memoryagentbench)                                         | Answer fact questions, resolve conflicting facts, answer from a 197K-token context and learn labels supplied at test time.                                                                                                 | Selected slices: 15 contexts and 1,500 questions—retrieval 500, conflict 800, RULER 100 and test-time learning 100.                                                                                                                                                | Contexts chunked through `memory.set_from_messages`; generated answers produced by `memory.ask`.                                                                                                                                                                                                               | SubEM and token F1 score answers, not raw retrieval; test-time learning uses label presence. No external reader or LLM judge.                                                                                                                                                                                                                             |
| [MuSiQue-Ans development sample](#musique-ans)                                        | Retrieve multiple supporting paragraphs for a multi-step question, then answer from those passages.                                                                                                                        | Seed-0 sample of 200 questions from the development set.                                                                                                                                                                                                           | Each question's paragraphs written with `memory.set`; semantic `memory.search` top 10; fixed `gpt-4o-mini` reader.                                                                                                                                                                                             | Mean fraction of gold supporting paragraphs retrieved at 10; exact match and token F1 against gold answers; no LLM judge.                                                                                                                                                                                                                                 |
| [PerLTQA sample](#perltqa)                                                            | Answer personal long-term-memory questions about profiles, relationships, events and dialogues.                                                                                                                            | 193-question slice: five fictional characters, at most 10 questions per section.                                                                                                                                                                                   | Character histories ingested with `memory.set_from_messages`; answered by `memory.ask`.                                                                                                                                                                                                                        | `gpt-4o-mini`-judged answer accuracy against reference answers, with two additional judges reported.                                                                                                                                                                                                                                                      |
| [Tracker duplicate detection](#duplicate-check-on-public-clickhouse-issues)           | Given a new issue draft, find its recorded older duplicate while limiting duplicate verdicts on issues with no recorded duplicate/related link.                                                                            | 104 recorded duplicate pairs and 100 negative-labelled issues from ClickHouse's public tracker.                                                                                                                                                                    | `tracker.duplicate_check` receives title and body as of creation time; the draft's issue number and recorded links are withheld.                                                                                                                                                                               | Older linked issue ranked first and duplicate verdict correctness; a negative label means no recorded link, not proof that two issues are unrelated.                                                                                                                                                                                                      |
| [ClickHouse code intelligence](#code-intelligence-on-the-clickhouse-repository)       | Find definitions, callers and implementation files for authored questions; abstain on authored out-of-repository probes.                                                                                                   | 60 authored questions: 10 definitions, 4 caller sets, 36 conceptual searches and 10 out-of-repository probes.                                                                                                                                                      | Public ClickHouse repository index; `code.symbol`, `code.callers` and `code.search` with their normal result limits.                                                                                                                                                                                           | Exact first definition; F1 against expected callers; mean gold-file recall@5/@10, MRR and hit@5; abstention precision/recall.                                                                                                                                                                                                                             |
| [SWE-bench Verified sample: file localization](#swe-bench-verified-file-localization) | Given a GitHub issue description, rank the pre-existing repository files that its accepted patch later modified. This is retrieval, not patch generation or issue resolution.                                              | Stratified seed-7 sample of 200 eligible instances from the 500-instance dataset, covering all 12 repositories; eligibility requires at least one patch-modified file present at `base_commit`.                                                                    | Each repository indexed at the issue's `base_commit`; issue text truncated at 2,000 characters; `code.search` first 10 results and a second page to 25 units.                                                                                                                                                  | File hit@1, hit@5 and hit@10 (at least one gold file); all-gold@10; MRR; paired against BM25 over the same live files and over Python files only.                                                                                                                                                                                                         |
| [Memory versus full history](#with-and-without-loom)                                  | Answer the same long-conversation question from retrieved memories or from the complete conversation history.                                                                                                              | Paired seed-13 sample of 100 LongMemEval-S questions.                                                                                                                                                                                                              | Same `gpt-5` reader sees either `memory.search` top 200 or the full history.                                                                                                                                                                                                                                   | Official-judge accuracy and reader prompt tokens; memory construction is excluded from prompt-token totals and reported separately as break-even.                                                                                                                                                                                                         |
| [Coding agents on ClickHouse: with Loom vs without Loom](#with-and-without-loom)      | Answer a ClickHouse developer's question: explain a mechanism, list the callers a change must update, find the change behind a behaviour, trace a bug to its fix, find the tests that cover a change.                      | 48 answer-graded questions × 3 paired runs and 36 conceptual questions × 10 paired runs; checkouts without git history at the verified commit.                                                                                                                     | Same headless `claude-sonnet-5` in a stock session; baseline has Bash, Read, Grep and Glob, the Loom arm adds the Loom plugin and its MCP `coding` toolset.                                                                                                                                                    | Judged key points of a verified reference answer, right file among the up to three the agent is asked to name; model cost, turns and wall time per run.                                                                                                                                                                                                   |
| [Definitions and callers versus grep](#with-and-without-loom)                         | Find a symbol's definition or all expected callers using Loom or repository text search.                                                                                                                                   | The same 10 definition and 4 caller questions from the ClickHouse code-intelligence set.                                                                                                                                                                           | Loom's code index versus `git grep` for the bare name and the qualified `Class::name`; grep output tokens estimated as characters divided by four.                                                                                                                                                             | Correct file ranked first, expected callers found and context tokens returned.                                                                                                                                                                                                                                                                            |
| [`code.ask` production cache](#with-and-without-loom)                                 | Measure how often repeated real questions reuse a cached answer and what one hit saves.                                                                                                                                    | 673 non-benchmark calls from one deployment over 30 days; 20 cache hits.                                                                                                                                                                                           | Normal `code.ask` traffic using the shipped cache guard and threshold.                                                                                                                                                                                                                                         | Observed hit rate, answer tokens avoided and latency; this is an operational observation, not a controlled quality comparison.                                                                                                                                                                                                                            |
| [`code.ask` cache threshold](#codeask-answer-cache-threshold)                         | Determine whether paraphrased questions can share an answer without sharing across materially different questions.                                                                                                         | 42 paraphrase pairs, 43 different-question pairs, 22 one-identifier near-misses and 2 unlabelled pairs.                                                                                                                                                            | About 130 embedding calls for each tested embedding model; cosine sweep with and without the identifier guard.                                                                                                                                                                                                 | Paraphrase hits versus false hits at each threshold; the shipped threshold requires zero false hits on the 65 labelled different pairs.                                                                                                                                                                                                                   |
| [Design decisions from design documents](#design-decisions-from-design-documents)     | Answer why questions, make change requests whose obvious move a design document rejected, answer questions on reversed or unrecorded decisions, and explain how a feature works, given different design knowledge per arm. | Private set: 5 design documents (4 revised once), 10 why, 15 change, 10 supersession or abstention and 13 feature tasks, 3 samples each.                                                                                                                           | Nothing, `knowledge.get`, knowledge hits of `loom_search`, gold rows or extracted rows through `memory.for_file` and `loom_search`, or extracted rows with the knowledge hits.                                                                                                                                 | `gpt-6-luna`-judged argument and feature-fact precision/recall, rejected-option avoidance, constraint satisfaction, stale statements and failed abstentions; paired against arm 0, the gold rows, and, for arm 6, the knowledge hits and the extracted rows alone.                                                                                        |
| [Document parsing](#document-parsing)                                                 | Read a PDF, DOCX or PPTX into text, headings, tables and page locators without losing content; then serve `knowledge.search` hits cited by page and quotes decision extraction can ground.                                 | Committed corpus of 54 files (6 public-domain PEPs, the rest synthetic), split dev and held-out by a fixed alternation within each category recorded in `gold.jsonl`; parse scores on the held-out 27; 212 search queries and 12 mined documents from both splits. | The reference extractor reading a PDF's text layer, and with its layout reader and OCR, and a layout-model baseline, through one normalizer; artifacts ingested through a stub extractor, fake n-gram embedder; `knowledge.search` top 5 within each document; decision extraction with `gpt-6-luna`, dry run. | Parse fidelity against gold labels from the specs the files were rendered from (text recall and precision, heading F1 and level, table cell F1, locators, silent loss) and efficiency (time and memory); retrieval (search found and cited at 1 and 5) and extraction quote match. Paired against the text layer with 95% bootstrap intervals over files. |
| [Memory at scale](#memory-at-scale)                                                   | Load synthetic memories straight into the stores, then run the memory ops through one server under each authority.                                                                                                         | Synthetic, offline: one namespace of 250,000 and 500,000 keys, and 200,000 keys Zipf-sized over 10,000 namespaces in 1,000 orgs; 2.4 versions per key.                                                                                                             | `memory.get`, `memory.set` and `memory.search` (top 10), 400 calls each, and 40 `memory.bulk_set` calls of 20 entries, at concurrency 5; `erasure.request` by user and by an org's namespaces; `memory.set` at fixed rates over several processes; fake 3072-dimension embedder.                               | p50 and p99 latency and refusals, ClickHouse bytes, CPU and memory per call from `system.query_log`, parts and merges from `system.part_log` and `system.parts`, store bytes and list-price cost per million memories; not a capacity promise (one shared host).                                                                                          |

These protocols cover the public measurements, not every runnable harness. The
[coverage matrix](/benchmarks/coverage) marks surfaces that currently have only a harness or no
public measurement, so a product capability is never presented as a measured result.

### Models

Every benchmark server ran with the deployment's model settings, recorded field by field in
`bench/results/models.json` (`python -m bench.models_used`), except where a section says its
run does not record them.

| Job                                                             | Model                            |
| --------------------------------------------------------------- | -------------------------------- |
| Embeddings (memories, queries, code chunks)                     | `text-embedding-3-large`, 3072-d |
| Extraction, rerank (`llm_model`)                                | `gpt-6-luna`                     |
| `memory.ask` strong tier                                        | `claude-opus-5-5`                |
| `memory.ask` cheap tier                                         | `gpt-6-luna`                     |
| `memory.ask` mid tier                                           | off                              |
| Durability check on coding captures                             | `claude-opus-5-5`                |
| Session briefs, duplicate-check rerank and judge, bisect rerank | `gpt-6-luna`                     |
| Retrieval judge                                                 | `gpt-6.1-sol`                    |
| `code.search` HyDE passage                                      | `gpt-6-luna`                     |
| `code.ask` synthesis                                            | `gpt-6.1-sol`                    |

The reader and the judge of each benchmark are the bench's own and are named in its
section; they are pinned independently of the server's models. A reasoning model in the
bench (reader or judge) runs at the lowest reasoning effort it accepts unless its section
says otherwise. SWE-bench file localization and the design-rationale bench name their own
models.

### LoCoMo conversational QA

All 10 LoCoMo conversations (1,986 questions) ingested once through
`memory.set_from_messages`, then answered two ways over the same memory: by an external
reader from `memory.search` hits (top 10, the paper protocol), and by Loom's own
`memory.ask`. Categories 1–4 are answerable and scored by an LLM judge. Category 5 is
adversarial: a response is correct only when it abstains, and published LoCoMo comparisons
normally exclude this category.

| Category             |     n | External reader | `memory.ask` |
| -------------------- | ----: | --------------: | -----------: |
| Single-hop           |   841 |           88.8% |        93.0% |
| Multi-hop            |   282 |           66.0% |        87.9% |
| Temporal             |   321 |           78.8% |        87.5% |
| Open-domain          |    96 |           43.8% |        71.9% |
| **Categories 1–4**   | 1,540 |       **79.7%** |    **89.6%** |
| Adversarial (cat. 5) |   446 |           78.9% |        90.8% |

On answerable questions the external reader abstained 7.8% of the time. Search latency p50
1,478 ms, p95 3,604 ms; a full question (search and answer) p50 2,054 ms, p95 4,291 ms.
Each conversation left about 265 memories.

- **Models:** the server's above (extraction `gpt-6-luna`; `memory.ask` strong
  `claude-opus-5-5`, cheap `gpt-6-luna`). External reader `gpt-4.1-mini`; judge `gpt-4.1`
  (the harness's calibration on 100 known-correct and 100 known-wrong answers: TPR 1.00,
  FPR 0.01).
- **Artifacts:** `bench/results/locomo_qa.{csv,summary.json}` (external reader),
  `bench/results/locomo_qa_ask.{csv,summary.json}` (`memory.ask`).
- **Reproduce:** `python bench/locomo_qa.py --dataset bench/datasets/locomo10.json --base-url <server> --token <token> --org <org> --keep-namespaces --namespaces-json <map.json> --out <a>.csv`,
  then `--reader ask --namespaces-json <map.json> --out <b>.csv` answers the same memory
  with `memory.ask`; `--gate --compare A.csv B.csv` pairs two runs. Judged accuracy moves a
  few points between runs ([methodology](/benchmarks/methodology)).

### LongMemEval-S

All 500 LongMemEval-S questions, each over its own haystack of about 47 sessions ingested
through `memory.set_from_messages`, then answered over that memory two ways: by Loom's own
`memory.ask`, and by a pinned external reader given `memory.search`'s top 200 hits. Both
are graded by the official LongMemEval judge.

The `memory.ask` column measures Loom's end-to-end product configuration, including its
configured answer models. It is not the fixed-reader configuration normally submitted to
the LongMemEval leaderboard and should not be compared directly with systems using a
different reader. The separate `gpt-5` column holds the reader fixed.

| Question type             |   n | `memory.ask` | Reader `gpt-5`, top 200 | Recall@30 |
| ------------------------- | --: | -----------: | ----------------------: | --------: |
| Single-session user       |  70 |        97.1% |                   98.6% |     98.6% |
| Single-session assistant  |  56 |       100.0% |                   91.1% |     94.6% |
| Single-session preference |  30 |       100.0% |                  100.0% |     90.0% |
| Knowledge update          |  78 |       100.0% |                   92.3% |    100.0% |
| Temporal reasoning        | 133 |        97.0% |                   94.0% |    100.0% |
| Multi-session             | 133 |        91.7% |                   85.0% |    100.0% |
| **All**                   | 500 |    **96.6%** |               **92.0%** | **98.6%** |

Recall@30 is a per-question hit rate: a question counts when at least one of its labelled
evidence sessions is represented among the first 30 returned memories. It is not the mean
fraction of all supporting sessions retrieved.

The same answers graded by two more judges with the official prompt: `gpt-6.1-sol` 97.8%,
`claude-opus-5-5` 97.6%. The re-grade's three judges, these two and `gpt-4o-2024-08-06`,
agree with each other on 98.8–99.8% of the questions.

The table's `memory.ask` figures are the `ask` labels in `longmemeval_s.csv`. The re-grade in
`longmemeval_s_judges.json` called `gpt-4o-2024-08-06` again on the same answers, and at
least two of its labels differ from the table's: on net, one multi-session answer it
counts correct (91.7% → 92.5%) and one preference answer it counts wrong (100% → 96.7%),
the total staying at 483 of 500 (96.6%). The three re-grading judges agree on every
label that differs from the table. The judge runs at temperature 0, which does not make
a hosted model's replies repeatable.

The `memory.ask` answers written during ingest scored 97.2%. The external reader runs at
reasoning effort `medium`; at `minimal` the same hits scored 86.0% (in
`longmemeval_s_ingest.csv`, arm `reader_gpt5_k200`). `memory.ask` answers are not the
pinned-reader figure LongMemEval publications report; the reader arm is.

`memory.search` at `top_k` 15, one call at a time over the 500 questions: p50 881 ms,
p95 2,687 ms, 1,722 tokens of context (p50); the first questions run on cold caches.
Repeated 100-question passes on warm caches measure p50 449–487 ms and p95 2.3–3.4 s.
Extraction used about 1.5 M tokens per question; each question has its own history.

- **Models:** the server's above (`memory.ask` strong `claude-opus-5-5`, cheap
  `gpt-6-luna`); external reader `gpt-5`; judge `gpt-4o-2024-08-06` with the official
  prompt, re-graded by `gpt-4o-2024-08-06`, `gpt-6.1-sol` and `claude-opus-5-5`.
- **Retrieval:** the `bench/longmemeval.py` searches (Recall@30 and the reader's hits)
  ran with `rerank` `builtin:openai` and `include_top_n_unmatched` 120, the harness
  defaults (the CSVs' `rerank` and `pool_unmatched` columns, which record the harness's own
  search on every row, the `ask` arm's included); `memory.ask` and the latency pass ran
  with no reranker.
- **Artifacts:** `bench/results/longmemeval_s.csv` (every verdict above),
  `longmemeval_s_ingest.csv` (the ingesting run: recall, token use, its own answers),
  `longmemeval_s_judges.json` (the three judges and their agreement) and
  `longmemeval_s_latency.json`.
- **Reproduce:**
  `python bench/longmemeval.py --limit 500 --shuffle --seed 13 --answer-judge --keep-namespaces --ask-arms '[{"label":"ask"},{"label":"reader_gpt5_k200","answerer":"reader","reader_model":"gpt-5","top_k":200}]' --base-url <server> --token <token> --out <file>.csv --hyp-out <file>.hyp.jsonl`
  with `LOOM_LLM_OPENAI_REASONING_EFFORT=medium` for the reader; `--reuse-csv <file>.csv`
  answers an ingested run again;
  `python -m bench.rejudge --hyp <file>.hyp.jsonl --dataset bench/datasets/longmemeval_s_cleaned.json --judge gpt-4o-2024-08-06 --judge gpt-6.1-sol --judge claude-opus-5-5`;
  `python -m bench.lme_search_latency --csv <file>.csv --base-url <server> --token-file <file> --top-k 15`.

### MemBench

Both MemBench scenarios, 20 questions of each type, answered by `memory.ask` and scored as
multiple-choice accuracy.

| Scenario      | Questions |  Accuracy |
| ------------- | --------: | --------: |
| Observation   |       140 | **0.879** |
| Participation |       140 | **0.879** |

Per type: observation single-hop 0.95, multi-hop 1.00, comparative 0.80, aggregative 0.70,
knowledge update 0.85, post-processing 0.95, preference 0.90; participation single-hop
0.95, multi-hop 0.90, comparative 0.80, aggregative 0.60, knowledge update 1.00,
post-processing 0.95, preference 0.90. One question is 0.05 of a type. Evidence recall at
ingest was 1.000 (observation) and 0.986 (participation).

- **Models:** the server's above; no bench reader or judge (multiple choice).
- **Artifacts:** `bench/results/membench_{observation,participation}.json` (the answers
  above) and `membench_{observation,participation}_ingest.json` (the ingesting run, with
  the evidence recall).
- **Reproduce:**
  `python bench/membench.py --data-dir <Membenchdata> --scenario observation --limit 20 --seed 13 --base-url <server> --token <token> --org <org> --out <file>.json`
  (and `--scenario participation`); `--namespace-json <map>` re-asks an ingested run.

### MemoryAgentBench

Selected MemoryAgentBench contexts were memorized chunk by chunk through
`memory.set_from_messages` and answered by `memory.ask`. These are generated-answer
measurements, despite the source split being named “Accurate Retrieval”; they are not raw
`memory.search` recall measurements. SubEM and token F1 are scored on the answer with
Loom's inline citation markers (`[memory_key]`) removed; they are provenance, not answer
words.

| Split                                      | Contexts | Questions |     SubEM |        F1 |
| ------------------------------------------ | -------: | --------: | --------: | --------: |
| Accurate retrieval (`eventqa_65536`)       |        5 |       500 | **0.990** | **0.957** |
| Conflict resolution (`factconsolidation_`) |        8 |       800 | **0.772** | **0.227** |
| Accurate retrieval (`ruler_qa1_197K`)      |        1 |       100 | **0.960** |     0.097 |
| Test-time learning (`icl_banking77`)       |        1 |       100 | **0.950** |         — |

Conflict resolution by context: single-hop 6k 0.93, 32k 0.94, 64k 0.92, 262k 0.89;
multi-hop 6k 0.77, 32k 0.71, 64k 0.56, 262k 0.46. Accurate retrieval on `eventqa_65536`
answered during ingest scored SubEM 0.994 and F1 0.928.

The RULER 197K context is one 197K-token document (497 chunks). Its answers state the
answer, then the supporting memories, so F1 against a short gold is low; counting only an
answer's first sentence, RULER scores 0.91. The test-time-learning score is the harness's
rule (the gold label appears in the answer); an answer often weighs several labels, and
counting only the first label it names scores 0.85.

- **Models:** the server's above; no bench reader or judge (SubEM, F1).
- **Artifacts:** `bench/results/memoryagentbench_ar.json` (the accurate-retrieval answers
  above), `memoryagentbench_ar_ingest.jsonl` (the ingesting run),
  `memoryagentbench_cr.jsonl`, `memoryagentbench_ruler197k.jsonl` and
  `memoryagentbench_ttl_banking77.jsonl`.
- **Reproduce:**
  `python bench/memoryagentbench.py --data <export.jsonl> --splits Accurate_Retrieval --source-regex eventqa_65536 --max-contexts 5 --max-questions 100 --memorize extract --answer ask --base-url <server> --token <token> --org <org> --out <file>.jsonl`
  (and `--splits Conflict_Resolution --source-regex factconsolidation_ --max-contexts 8`;
  `--splits Accurate_Retrieval --source-regex ruler_qa1_197K --max-contexts 1 --max-questions 0`;
  `--splits Test_Time_Learning --source-regex icl_banking77 --max-contexts 1 --max-questions 0`).

### MuSiQue-Ans

200 multi-hop questions sampled from the MuSiQue-Ans dev set (seed 0): each question's
paragraphs written through `memory.set`, retrieved by `memory.search` (semantic, top 10)
and answered by a pinned external reader. Recall@k is the mean fraction of each question's
gold supporting paragraphs present in the first `k` memories; it is not the share of
questions with any evidence hit.

| Measure   |     Value |
| --------- | --------: |
| EM        | **45.5%** |
| F1        | **55.3%** |
| Recall@10 | **0.890** |

By hops: recall@5 0.80 (2-hop), 0.77 (3-hop), 0.58 (4-hop). Search p50 982 ms.

- **Models:** embeddings `text-embedding-3-large` (no server LLM on this path); reader
  `gpt-4o-mini`; EM and F1 against the gold answer, no LLM judge.
- **Artifacts:** `bench/results/musique.{summary.json,jsonl}`.
- **Reproduce:**
  `python bench/musique.py --data musique_ans_v1.0_dev.jsonl --n 200 --seed 0 --top-k 10 --search-json '{"search_mode":"semantic"}' --reader --base-url <server> --token <token> --org <org> --out <file>.jsonl`.

### PerLTQA

The published sample of PerLTQA (5 characters, up to 10 questions per section: 193
questions): profile, social relationships, events and dialogues ingested through
`memory.set_from_messages`, answered by `memory.ask` and graded against the reference
answer.

| Section             |   n |  Accuracy |
| ------------------- | --: | --------: |
| Profile             |  43 |     0.954 |
| Social relationship |  50 |     0.880 |
| Events              |  50 |     0.920 |
| Dialogues           |  50 |     0.540 |
| **All**             | 193 | **0.819** |

The same answers graded by two more judges with the harness's prompt: `gpt-6.1-sol` 0.756,
`claude-opus-5-5` 0.839. Repeated runs (fresh ingests and re-asks) span 0.808–0.829, and
re-asking one store moves about 10 answers. Dialogues score lowest: their reference answers
often paraphrase the transcript ("the AI assistant mentioned private celebrations" for a
dinner a character arranged), and the grader rejects an answer that states the
transcript's own words.

- **Models:** the server's above; judge `gpt-4o-mini` (harness), re-graded by `gpt-6.1-sol`
  and `claude-opus-5-5` with the harness's prompt.
- **Artifacts:** `bench/results/perltqa.{summary.json,jsonl}`,
  `perltqa_judges.json`.
- **Reproduce:**
  `python bench/perltqa.py --mem perltmem_en_v2.json --qa perltqa_en_v2.json --max-characters 5 --max-questions-per-section 10 --namespace-json '{"default_conflict_policy":"newest_wins"}' --judge-model gpt-4o-mini --base-url <server> --token <token> --org <org> --out <file>.jsonl`.

### Duplicate check on public ClickHouse issues

`bench/tracker_dupcheck_bench.py` replays public ClickHouse issues closed as duplicates of an
older issue through `tracker.duplicate_check`, each as a new draft: its title and body,
`as_of` its creation time, and no issue number, so the op sees neither the item nor the
recorded link. 100 issues never closed as duplicates and carrying no duplicate or related
link are replayed the same way as negative-labelled examples. That label means the public
tracker has no recorded duplicate evidence; it does not prove the issues are unrelated.

The result is one tracker's. `tracker.duplicate_check` reads the tracker records bound to
the namespace (and the other tracker namespaces the caller may read), not the code index:
adding a repository indexes its code and merged PR history, never its issues, and a
repository with no tracker source gets no duplicate check. ClickHouse's public issues are
easier than many trackers' (stack traces, error codes, versions and SQL in the body, and
duplicates closed and linked consistently), so a team's own tracker can score differently;
the harness measures it on any tracker that serves the record contract's `github_issues`
and `edges` and records a duplicate as `state_reason = 'duplicate'` with a `related_to`
edge to the older item.

| Measure                                                |            Value |
| ------------------------------------------------------ | ---------------: |
| Older issue ranked first (104 duplicates)              |            0.875 |
| `duplicate_of` verdicts naming the right issue first   | 0.989 (90 of 91) |
| Verdicts on duplicates (duplicate_of / related / none) |      91 / 2 / 11 |
| No-recorded-link issues called a duplicate (100)       |             0.02 |
| Wall time p50 / p95                                    | 1,912 / 3,472 ms |

- **Models:** embeddings `text-embedding-3-large`; rerank `gpt-6-luna`; judge `gpt-6-luna`
  (`llm_model`); no bench judge (hit against the recorded duplicate link).
- **Artifacts:** `bench/results/tracker_dupcheck.json` (summary and every replayed
  draft's verdict and ranked numbers).
- **Reproduce:**
  `python -m bench.tracker_dupcheck_bench --base-url <server> --token-file <viewer token for the tracker namespace> --env-file <env with the tracker credential> --namespace <tracker namespace> --repo <owner/repo> --limit 200 --negatives 100 --out <file>.json`;
  `--namespace` and `--repo` point it at another tracker.

### Code intelligence on the ClickHouse repository

`bench/code_intel_qa.py` over its 60 authored questions (`bench/fixtures/code_intel_qa.jsonl`)
against an index of the public ClickHouse repository.

| Family                   |   n | Result                                              |
| ------------------------ | --: | --------------------------------------------------- |
| Symbol definitions       |  10 | exact@1 1.00                                        |
| Callers                  |   4 | F1 0.93                                             |
| Conceptual code search   |  36 | recall@5 0.91, recall@10 0.97, MRR 0.91, hit@5 1.00 |
| Out-of-repository probes |  10 | abstain precision 1.00, recall 0.60                 |

Answers returned a mean of 2,345 tokens (p90 2,930); `code.search` p50 was 2,799 ms. All 46
answers came from one index revision, which was behind the repository's newest commit, so
each carried the stale flag. Repeated runs move conceptual recall@5 by about 0.02.

For conceptual search, recall@k is the mean fraction of every question's labelled files
found in the first `k`; hit@5 asks only whether at least one labelled file appeared. Thus
the same run has recall@5 0.912 and hit@5 1.00—every question has a hit, but some questions
have additional labelled files outside the first five.

- **Models:** the server's above: embeddings `text-embedding-3-large`, `code.search`
  hypothetical-answer passage `gpt-6-luna`; no LLM judge.
- **Artifacts:** `bench/results/code_intel_qa.summary.txt` (the harness's
  summary) and `bench/results/code_intel_qa.jsonl` (every question and answer).
- **Reproduce:** `python bench/code_intel_qa.py --base-url <server> --token <viewer token> --org <org> --namespace <code namespace> --verbose --dump bench/results/code_intel_qa.jsonl`.
  The sets are small: one caller question moves the F1 by about 0.25.

### SWE-bench Verified file localization

This is a **file-localization benchmark**, not the official SWE-bench Verified
issue-resolution evaluation. Loom does not generate a patch, run the repository's tests
or receive credit for resolving an issue. The official benchmark evaluates generated
patches on a [human-validated set of 500 instances](https://www.swebench.com/verified.html).

`bench/swebench_localization.py` asks `code.search` with each instance's issue text
(`problem_statement`) and scores the files its gold `patch` modifies. No LLM judge. A
stratified sample of 200 eligible instances from the 500-instance dataset (seed 7, every
repository represented in proportion: django 91, sympy 29, sphinx 18, matplotlib 14,
scikit-learn 13, astropy 9, xarray 9, pytest 8, pylint 4, requests 3, seaborn 1, flask 1).
An instance is eligible when its accepted patch modifies at least one file that existed at
`base_commit`; instances whose patch only adds files are excluded because there is no
pre-existing file to retrieve.

| Arm                                      | hit@1 | hit@5 | hit@10 | all gold@10 |  MRR |
| ---------------------------------------- | ----: | ----: | -----: | ----------: | ---: |
| `code.search` (first 25 units, as files) |  0.44 | 0.765 |   0.82 |       0.755 | 0.58 |
| `code.search` first page only (10 units) |  0.44 |  0.77 |   0.82 |        0.75 | 0.57 |
| BM25, every indexed file                 |  0.19 |  0.45 |   0.57 |        0.50 | 0.31 |
| BM25, `.py` files only                   |  0.42 |  0.68 |   0.79 |        0.71 | 0.54 |

Paired over the 200 instances, `code.search` minus BM25 over the same files is +0.25
hit@10 (95% bootstrap interval [+0.18, +0.32]) and +0.27 MRR; against BM25 restricted to
`.py` files it is +0.035 hit@10 ([−0.025, +0.095]) and +0.03 MRR, not distinguishable.
37% of the top-5 units are prose (release notes, guides, changelogs) although every gold
file is source; a prose file ranks first on 49 of 200 issues. 57 issue texts are longer than 2,000 characters and were
sent cut to 2,000: hit@10 0.68 on those against 0.87
on the rest. `code.search` p50 1,939 ms, p95 4,462 ms; a first page returned a mean of
2,432 tokens. No answer abstained.

The table ran without the prose penalty, which ranks prose files below source unless the
question asks for documentation (`code_search_prose_penalty`, 0.25). Re-asked on the 12 instances whose repositories are
still indexed at the instance's commit, the gold file ranks first on 6 instead of 4, and
prose takes 8 instead of 27 of the 60 top-5 slots; hit@5 and hit@10 are unchanged (9 of 12).
Replaying the 200 stored rankings with prose scaled by 0.25 gives hit@1 0.53 and hit@5 0.81
(from 0.44 and 0.77), a lower bound since a replay cannot pull in files past the 25 units
it holds. The 36 conceptual questions above are unchanged by it (recall@10 0.963 without,
0.972 with).

- **Protocol:** gold = pre-image paths of files the patch modifies or deletes that exist
  at `base_commit`; patch-added files are excluded and `test_patch` is never read. One
  namespace per repository, backfilled with `--embed` at its oldest
  sampled `base_commit`, then moved commit to commit in commit-time order (deleted paths
  tombstoned, changed paths re-ingested with `loom code backfill --paths-file --commit`),
  so each query sees exactly its instance's tree. A step larger than 3,000 changed files
  starts a fresh backfill instead. Query: the issue text, cut to 2,000 characters;
  `top_k=10`, server-default bodies and legs, `fresh=true`; a second page through
  `next_cursor` deepens the ranking to 25 units. BM25 (k1 1.2, b 0.75, path plus content,
  identifier splitting) runs on the same query over the files the namespace holds live
  at that commit, and over the tree's `.py` files.
- **Models and configuration:** embeddings `text-embedding-3-large`, 512-dimension prefix
  shortlist, card and concept legs, hypothetical-answer passage written by `gpt-4o` (not the
  model in the table above); no chunk descriptions or file cards (a fresh
  backfill writes neither); one server against ClickHouse Cloud.
- **Cost:** 137.5 M embedding tokens (about $18 at list price) for 11 backfills and 188
  deltas; about 10.6 h of indexing over two repositories at a time.
- **Artifacts:** `bench/results/swebench_localization.summary.json` (every
  figure above, per repository, and the indexing cost) and
  `bench/results/swebench_localization.jsonl` (every instance: gold, ranked
  files of each arm, route, confidence, tokens, latency, index step).
- **Reproduce:** with a bench token and `OPENAI_API_KEY`,
  `python -m bench.swebench_localization run --sample 200 --seed 7 --base-url <server> --token-file <file> --env-file <the server's .env> --repos-dir <clones> --delta-mode repair --rebase-files 3000 --chain-files 1000000000 --out <rows.jsonl>`,
  then `python -m bench.swebench_localization bm25-py --rows <rows.jsonl> --repos-dir <clones>`
  and `python -m bench.swebench_localization report --rows <rows.jsonl>`. The run is
  resumable. At n=200 one instance moves a rate by 0.5 points.

### With and without Loom

What an agent or reader spends and gets with Loom against the same task without it. The
coding comparisons count the agent's own model bill; Loom's server-side model calls are not
included.

**Memory instead of the whole history (LongMemEval-S).** 100 questions (seed 13) answered
by `gpt-5` (reasoning effort `medium`) twice: from the whole chat history in the prompt,
and from `memory.search`'s top 200 hits over the ingested memory. Both are graded by the
official judge.

| Reader input            | Accuracy | Reader prompt tokens per question |
| ----------------------- | -------: | --------------------------------: |
| Whole history           |      90% |                           103,879 |
| Loom's top 200 memories |  **94%** |                         **9,443** |

Paired, 87 questions are right both ways, 7 only with memory, 3 only with the whole history.
The four-point accuracy difference is descriptive: only 10 outcomes disagree, and a
two-sided exact McNemar test gives p=0.34. This sample does not establish that retrieval is
more accurate than full history. Memory reads 91% fewer reader prompt tokens per question.
Those prompt-token totals exclude memory construction. Building the memory costs about
1.5 M extraction tokens per haystack, once, on `gpt-6-luna`. Against about 94 k
tokens saved per question, that is repaid after about 16 questions over the same history,
before the price gap between the extraction model and the reader is counted.

**Coding agents on ClickHouse, with and without Loom.** Claude Code (`claude-sonnet-5`) answers
each question twice in a stock session (no user or project settings): with Bash, Read, Grep and
Glob, and with the same tools plus the Loom plugin (hooks, and the MCP `coding` toolset with
`code.search` and `code.symbol` loaded up front). Sub-agents are off in both arms. Cost is the
Claude Code session's model bill (Loom's server-side model calls are not in it); every tool
call is counted per run, and a run that ends out of turns without an answer
counts as a miss at its full cost.

Each set runs on a checkout without git history at the commit the questions were
verified at (history questions keep full history), so neither arm can read a fix out of `git log`:

- **Answer-graded real work** (`bench/fixtures/code_work_v2.jsonl`, 24 questions, 3 paired runs):
  explain a mechanism end to end, list every caller and override a change must update, and find
  which merged change made a behaviour change. The agent answers in prose; `bench.agent_work_judge`
  (`claude-opus-5-5`) marks each key point of a reference answer verified in the code covered or
  not, and halves the score of a reply that contradicts it.
- **Conceptual questions** (`bench/fixtures/code_intel_qa.jsonl`, 36 questions, 10 paired runs):
  where a mechanism lives, scored by whether the right file is among the up to three files the
  agent is asked to end its reply with.
- **Multi-hop work** (`bench/fixtures/code_work_multihop.jsonl`, 24 questions, 3 paired runs): a
  bug report traced to its fix on a release branch whose fix was never backported (the branch
  checkout and the branch's own index), the tests that would catch a change, and setting- or
  error-to-call-site chains; answer-graded like the real-work set.

| Set                           | Quality, without → with Loom | Model cost per question, without → with | Turns, without → with |
| ----------------------------- | ---------------------------- | --------------------------------------- | --------------------- |
| Answer-graded real work (all) | judged score 0.78 → **0.84** | $0.212 → **$0.170** (−20%)              | —                     |
| — explain                     | 0.72 → **0.80**              | $0.312 → **$0.251** (−19%)              |                       |
| — impact                      | 0.88 → **0.92**              | $0.190 → **$0.148** (−22%)              |                       |
| — history                     | 0.73 → **0.81**              | $0.135 → **$0.110** (−18%)              |                       |
| Conceptual questions          | right file 91.4% → **96.4%** | $0.096 → **$0.076** (−21%)              | 5.5 → 5.7             |
| Multi-hop work (all)          | judged score 0.72 → **0.76** | $0.423 → **$0.396** (−6%)               | —                     |
| — tests that cover a change   | 0.76 → **0.81**              | $0.193 → **$0.154** (−21%)              |                       |
| — call chains                 | 0.87 → **0.89**              | $0.266 → **$0.223** (−16%)              |                       |
| — bug to fix, release branch  | 0.53 → **0.57**              | $0.811 → $0.811                         |                       |

Per question, the cost is compared against the same arm's run-to-run noise: two batches of the
without arm differ by more than 10% on a third of the questions at three runs each, so a question
counts as dearer with Loom only when the 90% bootstrap bound of its cost ratio stays above 1.
With Loom a question takes longer on the clock: the plugin's per-prompt search and the MCP
round trips add latency.

An earlier 12-issue SWE-bench agent sample is withdrawn: its with-Loom agent called no Loom tool.

**Definitions and callers against grep.** The 10 definition and 4 caller questions of
`bench/fixtures/code_intel_qa.jsonl`, replayed as `git grep` over a ClickHouse checkout:
once for the bare name, as an agent searching for a word would, and once for `Class::name`,
as a careful one would. Tokens are characters / 4.

| Question    | Bare-name grep, median           | `Class::name` grep, median    | Loom                                     |
| ----------- | -------------------------------- | ----------------------------- | ---------------------------------------- |
| Definitions | 29,061 tokens over 227 files     | 1,623 tokens over 25 files    | 1,678 tokens; right file first, 10 of 10 |
| Callers     | 558 tokens; 4 of 12 callers seen | 101 tokens; misses call sites | 12 of 12 expected callers                |

For a method with a distinctive qualified name, a careful grep is as cheap as Loom. For a
free function or a common type name (`executeQuery`, `parseQuery`, `Field`) grep returns
35 k to 490 k tokens where Loom returns about 1.2 k to 2.5 k. For callers grep is cheap but
incomplete: a call site spells the receiver, not the class (`reader.readFromParts(…)`), so
the qualified grep misses them and the bare grep names a third of them.

**Repeated `code.ask` questions.** Over 30 days of one deployment's real (non-bench)
traffic, 20 of 673 `code.ask` calls (3%) were served from the answer cache: each saved a
`gpt-6.1-sol` answer of about 3,700 tokens and took 385 ms instead of 7.8 s, about 74 k
tokens in all. The cache works but repeats are rare; how often it serves a wrong answer is
the threshold sweep below (0 of 65 different pairs at the shipped threshold).

- **Artifacts:** `bench/results/longmemeval_s_full_history.json` (every paired question),
  `agent_clickhouse_work.jsonl`, `agent_clickhouse_multihop.jsonl` and `agent_conceptual_ab.jsonl`
  (every run's arm, tokens, turns, Loom calls, cost, files, scores and judged key points),
  `code_grep_baseline.json` and `code_ask_cache_production.json`.
- **Reproduce:**
  `python -m bench.lme_full_context --dataset bench/datasets/longmemeval_s_cleaned.json --pair bench/results/longmemeval_s.csv --limit 100`
  (with `LOOM_LLM_OPENAI_REASONING_EFFORT=medium`);
  `python -m bench.agent_localization_ab --questions <fixture> --checkout <checkout at the pinned commit> --namespace <ns> --org <org> --mcp-url <server>/mcp?toolset=coding --token-file <file> --plugin-dir integrations/claude-code-plugin --code-repo ClickHouse/ClickHouse --unique-prompts --out <file>.jsonl`
  then `python -m bench.agent_work_judge --runs <file>.jsonl --fixture bench/fixtures/code_work_v2.jsonl --out <judged>.jsonl`;
  (`--session bare` for the up-front listing);
  `python -m bench.code_grep_baseline --checkout <ClickHouse checkout> --loom bench/results/code_intel_qa.jsonl`.

### `code.ask` answer-cache threshold

`code.ask` serves a cached answer when the nearest cached question clears a per-model
cosine threshold and passes the identifier guard (`guard_passes` in
`server/code/app/answers_cache.py`: both questions must name the same identifiers,
settings, paths, numbers, quoted strings and error codes). The sweep measures the trade
between cache hits and serving an answer to a different question.

| Artifact                                             | Content                                                                                                                                                                             |
| ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `bench/code_ask_cache_threshold/pairs.json`          | 42 `paraphrase` pairs (should share an answer), 43 `different` pairs (mostly the same subsystem), 22 `different_identifier` near-misses that differ in one identifier, 2 unlabelled |
| `bench/code_ask_cache_threshold/summary.json`        | The sweep under `text-embedding-3-large` (3072 dimensions)                                                                                                                          |
| `bench/code_ask_cache_threshold/summary_3small.json` | The sweep under `text-embedding-3-small` (1536 dimensions)                                                                                                                          |
| `bench/code_ask_cache_threshold/measure.py`          | The script that produced both                                                                                                                                                       |

Each summary records the embedding `model` and `dim`, not a commit or the guard's
version; `git log` on the file is the provenance.

**Reproduce** (about 130 embedding calls; needs `OPENAI_API_KEY`):

```bash
.venv/bin/python bench/code_ask_cache_threshold/measure.py --env-file .env \
  --model text-embedding-3-large --out /tmp/summary.json
.venv/bin/python bench/code_ask_cache_threshold/measure.py --env-file .env \
  --model text-embedding-3-small --out /tmp/summary_3small.json
diff <(jq -S . /tmp/summary.json) <(jq -S . bench/code_ask_cache_threshold/summary.json)
```

Embedding APIs are not bit-stable across provider revisions: expect cosines to agree to a
few thousandths. A threshold's margin must exceed that drift.

**With the guard on** (`sweep_with_guard`: paraphrase hits of 42 and false hits of 65
different pairs):

| Threshold | 3-large paraphrase hits | 3-large false hits | 3-small paraphrase hits | 3-small false hits |
| --------- | ----------------------- | ------------------ | ----------------------- | ------------------ |
| 0.70      | 36                      | 7                  | 36                      | 6                  |
| 0.75      | 30                      | 5                  | 29                      | 4                  |
| 0.80      | 29                      | 2                  | 24                      | 1                  |
| 0.85      | 14                      | 0                  | 17                      | 1                  |
| 0.90      | 6                       | 0                  | 5                       | 0                  |

**Recorded choices:**

| Field                  | Rule                                                                           | 3-large                                   | 3-small                       |
| ---------------------- | ------------------------------------------------------------------------------ | ----------------------------------------- | ----------------------------- |
| `chosen`               | Lowest 0.01 step above every `different` pair, cosine alone                    | 0.88: 12 of 42 paraphrases, margin 0.0003 | 0.87: 12 of 42, margin 0.0063 |
| `chosen_all_different` | Cosine alone, including the one-identifier near-misses                         | 0.99: 0 of 42                             | 0.99: 0 of 42                 |
| `chosen_with_guard`    | Lowest step with no different pair that clears the cosine and passes the guard | 0.84: 17 of 42, margin 0.0056             | 0.87: 11 of 42, margin 0.0063 |

Cosine alone cannot separate a question from its one-identifier twin (two issue numbers
one apart sit at 0.9857 under 3-large), so without the guard the only safe threshold
serves no paraphrase. The guard refuses all 22 near-misses and 12 of the 43 different
pairs, which lets the threshold drop to where a third of paraphrases hit.

**What ships.** `CACHE_HIT_THRESHOLDS` serves 0.85 for `text-embedding-3-large` (one step
above 0.84: 14 hits instead of 17, margin 0.0156 instead of 0.0056) and 0.87 for
`text-embedding-3-small`. An unlisted model takes 0.87; `LOOM_CODE_ASK_CACHE_THRESHOLD`
(> 0) overrides all. The guard cannot be turned off, so the unguarded columns are context.

**Limits.** One pair moves a hit rate by more than two points. The pairs come from one
deployment's cached questions over two repositories. Re-run the script for a new embedder
before serving the cache on it.

### Design decisions from design documents

`bench/design_rationale/` asks a coding agent four kinds of task about code its team's
design documents describe: why questions (A), change requests whose obvious move is an
option the documents rejected or a constraint they set (B), questions about a statement
a later revision reversed or about code no document covers (C), and questions about how
a feature works, its limits, compatibility and dependencies (D). Each arm shows the
agent different design knowledge: nothing (0), the whole document through
`knowledge.get` (1), the document as knowledge through the knowledge hits of
`loom_search` (2), hand-written gold decision rows through the pre-edit block and the
memory section of `loom_search` (3), the rows `loom decisions extract --apply` mined
from the same documents, through the same surfaces (4), and those extracted rows with
the documents' knowledge hits of `loom_search` (6). Arm 4 approximates the sync leg:
documents ingested as a document source's sync writes them, mined by the same extraction
function. The extractor in this run read 6,000-character windows and kept a decision
only when its reasoning and each rejected option's reason were one verbatim span of one
chunk; the whole-document extractor with cited, checked claims is not in these tables.
Extracted rows get files from a read-only code index of the ClickHouse repository, so
arms 4 and 6 read the pre-edit block too.

The task set is private: five design documents for the ClickHouse repository, four of them
revised once for the supersession tasks; 10 A, 15 B, 10 C (5 supersession, 5 abstention)
and 13 D tasks. The C revisions and abstention questions are synthetic. Each task is
answered 3 times per arm; each answer is judged once in each order.

| Arm | A precision | A recall | A misleading | A abstains | B avoids | B satisfies | C passes | C states current | D precision | D recall | D misleading | D abstains | Citation precision | Citation recall |
| --- | ----------: | -------: | -----------: | ---------: | -------: | ----------: | -------: | ---------------: | ----------: | -------: | -----------: | ---------: | -----------------: | --------------: |
| 0   |        29.3 |     30.1 |         78.3 |      100.0 |     18.9 |        12.2 |     93.3 |             13.3 |        15.6 |     12.1 |         65.4 |      100.0 |                  - |             0.0 |
| 1   |        84.1 |     92.1 |         25.0 |       20.0 |      4.4 |         2.2 |    100.0 |            100.0 |        89.5 |     76.0 |         23.1 |        0.0 |               83.4 |            79.0 |
| 2   |        81.3 |     80.6 |         41.7 |       26.7 |     42.2 |        43.3 |    100.0 |            100.0 |        80.3 |     59.7 |         47.4 |       10.3 |               54.5 |            48.9 |
| 3   |        96.8 |     88.8 |          8.3 |       10.0 |     87.8 |        81.1 |    100.0 |            100.0 |        91.9 |     65.5 |         16.7 |        6.4 |               94.0 |            94.2 |
| 4   |        53.6 |     32.9 |         36.7 |       63.3 |     54.4 |        52.2 |     96.7 |             10.0 |        65.0 |     26.9 |         37.2 |       69.2 |               92.1 |            63.8 |
| 6   |        84.8 |     83.9 |         26.7 |       20.0 |     55.6 |        55.6 |    100.0 |            100.0 |        85.4 |     67.7 |         42.3 |        5.1 |               84.0 |            74.9 |

Percentages. A and D precision is the share of an answer's claims that state a gold
argument (A) or gold fact (D) or what a gold row states, over the answers that made a
claim; recall is the share of the gold arguments or facts the answer states; A and D
abstain are the share of answers saying no recorded decision (A) or design (D) covers the
question, in whole or in part; A and D misleading are the share of answers that would
steer an engineer wrong. B avoids: no rejected option proposed (deterministic detectors
first, the judge for the rest) and the rubric passed where the task has one; B satisfies:
no gold constraint broken, no rejected option detected, and the rubric passed where the
task has one. C passes: a supersession answer does not present a reversed statement as the
current design, an abstention answer says no decision is recorded and invents none. C
states current is averaged over the 5 supersession tasks. A reversed statement was
presented as current in 6.7% of the supersession answers in arms 0 and 4, none in the
other arms; 6.7% of the abstention answers failed in arm 0, none in the other arms.
Citation precision is the share of the answers' citations that resolve and that the judge
says support what they are cited for. Citation recall is the share of A and D claims with
at least one of their own citations resolving and supporting them. Answers with no claim,
which precision leaves out: of the A answers 6.7% in arm 4, of the D answers 10.3% in arm
0 and 17.9% in arm 4, none in the other arms.

Paired per task, bootstrap over documents, 95% interval, and the minimum detectable effect
taken from that interval. With five clusters the bootstrap intervals and the MDEs taken
from them understate the uncertainty:

| Treatment vs reference | Kind | Metric    | n (clusters) | Delta | 95% interval   |  MDE |
| ---------------------- | ---- | --------- | -----------: | ----: | -------------- | ---: |
| 3 vs 0                 | A    | precision |       10 (5) | +67.5 | +61.0 .. +82.7 | 15.5 |
| 3 vs 0                 | A    | recall    |       10 (5) | +58.7 | +53.1 .. +73.8 | 14.8 |
| 3 vs 0                 | B    | avoids    |       15 (5) | +68.9 | +59.5 .. +86.7 | 19.4 |
| 3 vs 0                 | B    | satisfies |       15 (5) | +68.9 | +40.9 .. +89.4 | 34.6 |
| 3 vs 0                 | C    | passed    |       10 (5) |  +6.7 | +0.0 .. +12.1  |  8.7 |
| 3 vs 0                 | D    | precision |       13 (5) | +76.3 | +60.5 .. +87.2 | 19.0 |
| 3 vs 0                 | D    | recall    |       13 (5) | +53.3 | +41.0 .. +71.6 | 21.8 |
| 4 vs 0                 | A    | precision |       10 (5) | +24.3 | -9.5 .. +45.8  | 39.5 |
| 4 vs 0                 | A    | recall    |       10 (5) |  +2.8 | -18.1 .. +16.2 | 24.4 |
| 4 vs 0                 | B    | avoids    |       15 (5) | +35.6 | +0.0 .. +59.5  | 42.5 |
| 4 vs 0                 | B    | satisfies |       15 (5) | +40.0 | +8.3 .. +61.1  | 37.7 |
| 4 vs 0                 | C    | passed    |       10 (5) |  +3.3 | +0.0 .. +8.3   |  5.9 |
| 4 vs 0                 | D    | precision |       11 (5) | +46.5 | +1.3 .. +69.3  | 48.6 |
| 4 vs 0                 | D    | recall    |       13 (5) | +14.7 | -0.7 .. +22.4  | 16.5 |
| 4 vs 3                 | A    | precision |       10 (5) | -43.2 | -81.0 .. -22.9 | 41.5 |
| 4 vs 3                 | A    | recall    |       10 (5) | -55.9 | -83.3 .. -39.4 | 31.4 |
| 4 vs 3                 | B    | avoids    |       15 (5) | -33.3 | -64.1 .. -11.9 | 37.3 |
| 4 vs 3                 | B    | satisfies |       15 (5) | -28.9 | -61.5 .. -8.3  | 38.0 |
| 4 vs 3                 | C    | passed    |       10 (5) |  -3.3 | -10.0 .. +0.0  |  7.1 |
| 4 vs 3                 | D    | precision |       11 (5) | -29.2 | -59.5 .. -15.1 | 31.7 |
| 4 vs 3                 | D    | recall    |       13 (5) | -38.6 | -56.3 .. -28.2 | 20.1 |
| 1 vs 0                 | A    | precision |       10 (5) | +54.8 | +47.9 .. +69.5 | 15.4 |
| 1 vs 0                 | A    | recall    |       10 (5) | +62.0 | +48.7 .. +88.1 | 28.1 |
| 1 vs 0                 | B    | avoids    |       15 (5) | -14.4 | -22.0 .. -3.0  | 13.5 |
| 1 vs 0                 | B    | satisfies |       15 (5) | -10.0 | -15.2 .. -1.5  |  9.7 |
| 1 vs 0                 | C    | passed    |       10 (5) |  +6.7 | +0.0 .. +12.1  |  8.7 |
| 1 vs 0                 | D    | precision |       13 (5) | +73.9 | +59.0 .. +82.1 | 16.4 |
| 1 vs 0                 | D    | recall    |       13 (5) | +63.8 | +54.7 .. +75.3 | 14.7 |
| 1 vs 3                 | A    | precision |       10 (5) | -12.7 | -20.2 .. -6.9  |  9.5 |
| 1 vs 3                 | A    | recall    |       10 (5) |  +3.3 | -7.1 .. +22.5  | 21.2 |
| 1 vs 3                 | B    | avoids    |       15 (5) | -83.3 | -94.0 .. -70.0 | 17.2 |
| 1 vs 3                 | B    | satisfies |       15 (5) | -78.9 | -97.0 .. -42.4 | 39.0 |
| 1 vs 3                 | C    | passed    |       10 (5) |  +0.0 | +0.0 .. +0.0   |  0.0 |
| 1 vs 3                 | D    | precision |       13 (5) |  -2.4 | -10.6 .. +3.6  | 10.2 |
| 1 vs 3                 | D    | recall    |       13 (5) | +10.5 | +4.4 .. +15.5  |  7.9 |
| 2 vs 3                 | A    | precision |       10 (5) | -15.6 | -27.5 .. -7.6  | 14.2 |
| 2 vs 3                 | A    | recall    |       10 (5) |  -8.2 | -17.6 .. +8.5  | 18.7 |
| 2 vs 3                 | B    | avoids    |       15 (5) | -45.6 | -52.8 .. -40.2 |  9.0 |
| 2 vs 3                 | B    | satisfies |       15 (5) | -37.8 | -52.8 .. -19.1 | 24.1 |
| 2 vs 3                 | C    | passed    |       10 (5) |  +0.0 | +0.0 .. +0.0   |  0.0 |
| 2 vs 3                 | D    | precision |       13 (5) | -11.6 | -14.1 .. -7.5  |  4.7 |
| 2 vs 3                 | D    | recall    |       13 (5) |  -5.8 | -13.8 .. +1.9  | 11.2 |
| 6 vs 2                 | A    | precision |       10 (5) |  +3.5 | -0.4 .. +10.3  |  7.6 |
| 6 vs 2                 | A    | recall    |       10 (5) |  +3.3 | -7.4 .. +8.6   | 11.4 |
| 6 vs 2                 | B    | avoids    |       15 (5) | +13.3 | -0.9 .. +38.3  | 28.0 |
| 6 vs 2                 | B    | satisfies |       15 (5) | +12.2 | -3.0 .. +38.3  | 29.5 |
| 6 vs 2                 | C    | passed    |       10 (5) |  +0.0 | +0.0 .. +0.0   |  0.0 |
| 6 vs 2                 | D    | precision |       13 (5) |  +5.1 | -4.1 .. +9.9   | 10.0 |
| 6 vs 2                 | D    | recall    |       13 (5) |  +8.0 | +1.3 .. +15.4  | 10.1 |
| 6 vs 4                 | A    | precision |       10 (5) | +31.1 | +13.5 .. +68.3 | 39.1 |
| 6 vs 4                 | A    | recall    |       10 (5) | +51.0 | +31.9 .. +80.2 | 34.4 |
| 6 vs 4                 | B    | avoids    |       15 (5) |  +1.1 | -27.3 .. +38.9 | 47.3 |
| 6 vs 4                 | B    | satisfies |       15 (5) |  +3.3 | -25.0 .. +40.3 | 46.6 |
| 6 vs 4                 | C    | passed    |       10 (5) |  +3.3 | +0.0 .. +10.0  |  7.1 |
| 6 vs 4                 | D    | precision |       11 (5) | +22.4 | +9.6 .. +44.8  | 25.2 |
| 6 vs 4                 | D    | recall    |       13 (5) | +40.8 | +29.1 .. +57.1 | 20.0 |
| 6 vs 3                 | A    | precision |       10 (5) | -12.1 | -20.1 .. -5.4  | 10.5 |
| 6 vs 3                 | A    | recall    |       10 (5) |  -4.9 | -13.0 .. +8.1  | 15.0 |
| 6 vs 3                 | B    | avoids    |       15 (5) | -32.2 | -44.7 .. -10.0 | 24.8 |
| 6 vs 3                 | B    | satisfies |       15 (5) | -25.6 | -44.7 .. +10.0 | 39.1 |
| 6 vs 3                 | C    | passed    |       10 (5) |  +0.0 | +0.0 .. +0.0   |  0.0 |
| 6 vs 3                 | D    | precision |       13 (5) |  -6.5 | -17.6 .. +0.4  | 12.8 |
| 6 vs 3                 | D    | recall    |       13 (5) |  +2.2 | -7.7 .. +10.2  | 12.8 |

The n of a D precision comparison with arm 4 is 11: on two tasks no arm 4 answer made a
claim to grade.

| Arm | Injected tokens per task (mean, max) | Pre-edit calls | Pre-edit max | Answers' cost, 144 answers (USD) |
| --- | -----------------------------------: | -------------: | -----------: | -------------------------------: |
| 0   |                                 0, 0 |              0 |            0 |                            0.011 |
| 1   |                         1,878, 3,432 |              0 |            0 |                            0.062 |
| 2   |                           725, 1,015 |              0 |            0 |                            0.033 |
| 3   |                         1,190, 2,758 |            129 |          663 |                            0.035 |
| 4   |                           296, 1,048 |            129 |          273 |                            0.015 |
| 6   |                         1,049, 1,790 |            129 |          273 |                            0.038 |

Arm 1 injects 1.58 times arm 3's tokens. By ledger stage the run spent 0.194 USD on
answers, 0.391 on 1,728 judge passes and 0.004 on the server's own model and embedding
calls. For extraction this run's ledger charged an upper bound on the extractor it ran,
once per document before its extraction run, 0.173 USD in all: the extraction's model
calls cost at most 0.173, and its write embeddings are not recorded. For the
whole-document extractor the harness charges an estimate, not a bound
(`bench/design_rationale/README.md`). The ledger does not split spend by arm: the answers'
cost per arm is their recorded tokens priced at the current price table, and judging is
not split.

**Reading.** The results this run is about; a result is resolved when its interval clears
zero (the lower bound above it, or the upper bound below it), and the comparison table
holds the rest. The gold rows (arm 3) raise A and D precision and recall and B avoidance
and satisfaction over no design knowledge. The whole document (arm 1) raises D recall 10.5
points over the gold rows and lowers B avoidance below arm 0 (-14.4). Extracted rows alone
(arm 4) fall below the gold rows on A, B and D; adding the documents' knowledge hits to
them (arm 6) raises A precision (+31.1) and recall (+51.0) and D precision (+22.4) and
recall (+40.8) over arm 4, and D recall over the knowledge hits alone (arm 2, +8.0), and
leaves B avoidance unresolved against arm 4 (+1.1, -27.3 .. +38.9) and 32.2 points below
the gold rows. Arms 3 and 6 are not resolved apart on D recall (+2.2, -7.7 .. +10.2).
Unpaired, as per-arm rates: answers mislead more often with the documents alone than with
the gold rows, D misleading 47.4 in arm 2 and 23.1 in arm 1 against 16.7 in arm 3, and A
misleading 41.7 and 25.0 against 8.3; with the extracted rows, D answers mislead at about
the same rate with or without the documents (42.3 in arm 6, 37.2 in arm 4), and A answers
less often with them (26.7 against 36.7). A hypothesis, not tested here: B avoidance
follows the rows the pre-edit block serves. Arm 4's and arm 6's block served a row on 9 of
the 43 tasks with gold rows, arm 3's on 39, but the arms also differ in the rows
themselves, and no arm varies the block's reach alone.

**Treatment check.** Of the 43 tasks with gold rows, arm 3's pre-edit block served 39 and
search 42 (a task counts when it served one of its own gold rows); arm 4's and arm 6's
pre-edit block served 9 and search 33 (a task counts when it served any extracted row).
The knowledge hits held the task's own document on all 43 tasks with gold rows in arms 2
and 6; with five documents and five hits shown, this shows the hits were served, not how
well they ranked. Extraction wrote 19 rows across the documents and their revisions, 12
with files, with no extraction failure and no failed code lookup.

**Judge stability.** The headline verdict (A and D misleading, B avoids, C passed) agreed
across the two orders on 90.0% of 827 answers; an answer whose compared pass is
unreadable, and a change answer a detector decided, are left out. Each answer was judged
once per order, so agreement across repeats is not measured. 49 of 1,728 judge passes were
unreadable (13 answers unscored).

**Limits.** One reader and one judge model; no second judge and no human-labelled slice,
so judge accuracy and kappa are not measured. Extraction precision is not measured (no row
was labelled). No patch is applied and no test runs, so change-task pass rate is not
measured. The task set is private: the numbers are reproducible only by a holder of the
set.

- **Models:** reader, judge and extraction `gpt-6-luna`, in the summary's run flags; the
  extraction model is the LLM model the server settings of the harness's environment
  resolve (`LOOM_LLM_MODEL`), and `run` refuses an `--extract-model` that differs. The run
  does not record the server's own models, nor the revision of the code index arms 4 and
  6 looked their code names up in. The server ran from source on its own ClickHouse, with
  auth and background workers off.
- **Artifacts:** `bench/results/design_rationale.summary.json`, written by
  `score --summary`: the per-arm metrics, comparisons, injected tokens, treatment check,
  judge agreement, unreadable passes and the A and D answers with no claim, the task
  set's counts (documents, revisions, revised documents, tasks by kind and subtype, tasks
  with gold rows), the run's flags (arms, whether the extracted arms had files, models,
  window, K, judge orders and repeats) and its spend by ledger stage and the answers'
  cost per arm; the per-task rows, which name private tasks, are left out.
- **Reproduce:**
  `PYTHONPATH=$PWD .venv/bin/python -m bench.design_rationale run <taskset> --out <dir> --base-url <server> --arms 0,1,2,3,4,6 --reader-model gpt-6-luna --judge-model gpt-6-luna --code-index <url> --code-index-org <org> --code-index-namespace <code namespace> --extract-model gpt-6-luna --k 3 --judge-repeats 1 --max-usd 3`,
  then `... score <taskset> --out <dir> --summary bench/results/design_rationale.summary.json`.

#### Noise sections next to the design documents

The same task set with ten distractor documents added: for each of the five features, the
notes of a team sync and a status report that name its components and discuss options the
design rejected, written for the benchmark. Three of them record a decision no design
document states, and three D tasks ask about those decisions. Arms 2 and 6 read the
same namespaces, the same documents and the same extracted rows twice, with
`LOOM_KNOWLEDGE_NOISE_FILTER` on and off; A, B and D tasks, answered 3 times per arm,
each answer judged once.

| Arm | Filter | A precision | A recall | A misleading | B avoids | D precision | D recall | D misleading | Notes decisions: D recall |
| --- | ------ | ----------: | -------: | -----------: | -------: | ----------: | -------: | -----------: | ------------------------: |
| 2   | off    |        75.7 |     81.1 |         41.7 |     35.6 |        66.8 |     51.6 |         74.4 |                      66.7 |
| 2   | on     |        76.3 |     77.3 |         38.3 |     26.7 |        80.1 |     50.0 |         57.7 |                      66.7 |
| 6   | off    |        89.8 |     86.8 |         26.7 |     57.1 |        80.9 |     59.0 |         52.6 |                      66.7 |
| 6   | on     |        85.2 |     92.6 |         45.0 |     66.7 |        84.9 |     66.8 |         16.7 |                      66.7 |

D columns are the 13 design-document tasks; the last column is the three tasks about
decisions recorded in notes, with no misleading answer in any row. Filter on minus off,
paired per task, bootstrap over documents, 95% interval:

| Arm | Kind | Metric     | Delta | 95% interval   |
| --- | ---- | ---------- | ----: | -------------- |
| 2   | A    | precision  |  +0.7 | -9.5 .. +8.9   |
| 2   | A    | recall     |  -3.8 | -8.1 .. +1.7   |
| 2   | A    | misleading |  -3.3 | -16.7 .. +14.3 |
| 2   | B    | avoids     |  -8.9 | -41.7 .. +19.0 |
| 2   | D    | precision  | +13.3 | +5.3 .. +26.7  |
| 2   | D    | recall     |  -1.6 | -7.2 .. +4.5   |
| 2   | D    | misleading | -16.7 | -50.0 .. -0.8  |
| 6   | A    | precision  |  -4.6 | -8.9 .. +4.0   |
| 6   | A    | recall     |  +5.8 | -0.9 .. +21.8  |
| 6   | A    | misleading | +18.3 | -11.9 .. +45.2 |
| 6   | B    | avoids     |  +9.5 | +0.0 .. +23.1  |
| 6   | D    | precision  |  +4.0 | -4.4 .. +17.5  |
| 6   | D    | recall     |  +7.8 | -0.2 .. +18.1  |
| 6   | D    | misleading | -35.9 | -70.8 .. -16.7 |

With the filter on, 15 of the 65 knowledge hits an agent read came from the distractor
documents (their decision sections, or noise no close design hit outranked), against 47
of 90 with it off; every design-document hit served with the filter off was served with
it on. A misleading rose for arm 6, inside its interval: a lean against the filter on
why-questions that this task set cannot resolve.

Reproduce: the run above with `--arms 2,6 --judge-orders 1 --keep-namespaces` against a
server started with `LOOM_KNOWLEDGE_NOISE_FILTER=true`; then a copy of that run's
`state.json` in a new `--out`, the server restarted with `LOOM_KNOWLEDGE_NOISE_FILTER=false`,
and the same command on the new `--out`, which answers over the same namespaces without a
second setup.

The labeller's own check (`tests/unit/test_section_value.py`) is a synthetic set of 141
labelled sections from design documents, meeting notes, status reports and templates,
written by a writer who did not see the rules: on the half the rules were written from,
noise precision 1.00 and recall 0.19 with all 10 decisions in notes kept; on the held-out
half, which was checked after each rule fix and so no longer measures the rules blind,
precision 1.00 and recall 0.17 with all 10 kept. The rules leave a section as
content when unsure, since a missed noise section only ranks higher than it should while
a design section labelled noise leaves a context pack: outside documents titled as
dated meeting notes or status reports, a section is noise only as an open-questions
list, so the set's templates, chatter, logistics and bookkeeping elsewhere are missed.

### Document parsing

`bench/docparse/` scores the reference extractor on a committed corpus of the documents Loom
ingests: 54 PDF, DOCX and PPTX files (browser-printed and office-exported design docs,
typeset specs, two-column and rotated pages, borderless tables and tables over pages, slide
decks and their exports, scanned pages, a whiteboard photo, and six public-domain PEPs),
split into dev and held-out within each category. Gold labels come from the specs the files
were rendered from. Three systems, each through the same normalizer, each named by the stamp
its parses carry: the reference extractor at PDF parser version 1, DOCX 2 and PPTX 2 (text
layer: a PDF's text layer only, a DOCX's headings at their outline levels, no OCR); the
reference extractor at version 3 of each, with OCR on (layout and OCR: a PDF read as laid
out, a DOCX's headings under its title, Tesseract 5.5 reading scanned pages and large
pictures); and a layout-model baseline (a document parser that runs neural layout and table
models on CPU, whose version and settings the parses do not record). Every parser change was
tuned on dev; the parse scores below are held-out (27 files).

| Measure (held-out, 27 files)    | Text layer | Layout and OCR | Baseline |
| ------------------------------- | ---------: | -------------: | -------: |
| Text recall                     |      0.876 |          0.999 |    0.915 |
| Text precision                  |      0.880 |          0.984 |    0.961 |
| Heading F1                      |      0.342 |          0.962 |    0.957 |
| Heading level accuracy          |      0.700 |          1.000 |    0.357 |
| Table cell F1                   |      0.273 |          0.991 |    0.998 |
| Page and slide locators         |      0.814 |          0.991 |    0.875 |
| Speaker notes as notes (3 PPTX) |      1.000 |          1.000 |    0.000 |
| Silent losses (files)           |          5 |              2 |       10 |
| Seconds a file (mean)           |      0.053 |          0.520 |    9.016 |
| Peak memory a file (MB)         |         37 |             65 |    1,592 |

Heading level accuracy is taken over the files where a system matched a heading: 9 for the
text layer, 26 for the others. Silent loss is a file reported complete that dropped a block,
a page's text or a tenth of its words; it is a count, not a mean. The layout and OCR
reader's two are OCR misreads on files reported complete: a YAML code block on a scanned
page, and one label of a blurred whiteboard photo on a slide. Its text recall over the text
layer is +0.122 [+0.015, +0.264] and its text precision +0.105 [+0.001, +0.237] (paired, 95%
bootstrap over files); every metric's interval is in the artifact.

By category, the layout and OCR reader matches or beats the baseline on every measure above,
except:

- **Scanned pages and the whiteboard photo (3 files; headings and tables over the 2
  scans):** heading F1 0.505 against 0.909 and table cell F1 0.928 against 0.978. OCR's
  point sizes separate a title from body text but not one heading level from the next, and a
  scanned grid drawn in light gray is not always found. Text recall (0.997 against 0.649)
  and locators (0.983 against 0.625) are higher.
- **PDF exports (5 files):** table cell F1 0.991 against 1.000.
- **Tables over pages (2 files):** locators 0.977 against 1.000.
- **Slides and PPTX (6 files):** text precision 0.960 and 0.951 against 1.000, as text OCR
  reads in a slide's pictures holds words the gold does not; and a PPTX takes 0.39 s a file
  against 0.12 s, the cost of reading each large picture with OCR, which brings PPTX text
  recall to 1.000 (baseline 0.696).

Overall the layout and OCR reader is 17 times faster than the baseline and uses 4% of its
peak memory. Each system's time is its parse of a file, timed inside the process that parses
it, its start-up aside; the baseline's models also load in each such process (median 3.2 s,
`load_seconds` in its parses, over 54 files). The extractor image is 511 MB with OCR and 467
MB without (OCR adds 44 MB unpacked); the baseline's image is 3.46 GB
(`bench/results/docparse_images.json`, `docker image ls` on each build).

**Downstream.** Each system's artifacts are ingested into a running Loom through a stub
extractor, and up to three paragraphs and two table rows of every file in both splits (212
queries) go to `knowledge.search` within their own document and system's namespace. It runs
under fake providers by design: the embedder and the LLM provider are fakes, the same for
every system (`docparse_models_downstream.json`), so the table ranks parses; it does not
measure Loom's retrieval under a real embedder. Found: a top-5 hit holds 70% of the answer's
words; cited: that hit's locator holds the answer's page. Ingests that failed score zero:
the text layer's four scans (no text without OCR). The whiteboard photo, whose ingest failed
for the text layer and the baseline, holds no query. The text layer and layout and OCR are
served their extractor's own artifacts, the baseline the artifacts `artifacts` builds from
its normalized parses.

| Measure                           | Text layer | Layout and OCR | Baseline |
| --------------------------------- | ---------: | -------------: | -------: |
| Cited at 1 (212)                  |      0.792 |          0.825 |    0.858 |
| Cited at 5 (212)                  |      0.887 |          0.943 |    0.948 |
| Found at 5 (212)                  |      0.920 |          0.976 |    1.000 |
| Cited at 5 (104 held-out queries) |      0.856 |          0.923 |    0.952 |

Against the text layer, layout and OCR is +0.057 [-0.029, +0.160] cited at 5 and +0.057
[-0.018, +0.149] found at 5 over all 212 queries; neither interval excludes zero. Scanned
pages go from 0 to 1.00 cited at 5. It loses significantly on rotated pages (cited at 1,
-0.30 [-0.40, -0.20]), borderless tables (cited at 1, -0.15 [-0.20, -0.05]) and tables over
pages (cited and found at 5, -0.20 [-0.35, -0.05]), and on tables over pages it trails the
baseline (cited at 1 0.60 against 0.95, found at 5 0.80 against 1.00). It is the only system
that misses queries whose file it ingested: 5 found in no top-5 hit, four rows of long
tables and one row of a two-column page's table. Where it loses at 5, a row is either found
and not cited, because the chunk that holds a table's first rows also holds the paragraph
before the table and so starts on the page before (the scorer cites a row only within its
table's pages), or not found, as `knowledge.search` folds the row blocks of one section and
a row deep in a long table falls out of the top five. At 1 it also loses some paragraph
queries of PDF exports to rank.

**Decision extraction.** The twelve PDFs of `DECISION_FILES` (six PEPs and six design docs,
from both splits) are mined in a dry run from each system's ingested text with `gpt-6-luna`,
and counted where extraction checks quotes against the parsed text:

| Measure (12 documents)                  | Text layer | Layout and OCR | Baseline |
| --------------------------------------- | ---------: | -------------: | -------: |
| Decisions, reasons and rejected options |         66 |             84 |       62 |
| Of them, quotes in the parsed text      |      0.985 |          1.000 |    0.984 |
| Kept quotes found in the gold text      |      1.000 |          1.000 |    0.971 |

One run of one model; the first count moves from run to run, so only the two rates are
comparable across systems.

- **Models:** no LLM or embedder for the parse scores (layout and OCR runs Tesseract 5.5,
  stamped in its parses; the baseline its own layout and table models); the downstream run
  used the fake n-gram embedder and the fake LLM provider
  (`bench/results/docparse_models_downstream.json`); decision extraction used `gpt-6-luna`
  (`bench/results/docparse_models_decisions.json`).
- **Artifacts:** `bench/results/docparse_parses/` (each system's normalized parse of every
  file: `text_layer.jsonl`, `layout_ocr.jsonl`, `baseline.jsonl`),
  `bench/results/docparse_heldout.json` (every metric per category with paired intervals,
  and every file's scores), `bench/results/docparse_downstream.json` (the `downstream`
  harness's output: ingest states, the query set with each answer, each query's judgement,
  the paired report by category and by split, and the decision counts; its systems renamed
  from the run's names, its queries given their split from `gold.jsonl`, and its report
  recomputed with the split groups). The baseline's seconds for typeset_1 and scan_2 time
  an earlier rendering of those files; their quality scores are of the committed files.

**Reproduce.** The parse scores are re-scored from the committed parses (the baseline's
column is not regenerated here). The `run` lines need an interpreter with the extractor's
dependencies (`--python` names one), and the layout-and-OCR line Tesseract 5.5 and its
English model as the extractor image installs them; without them its parses have no OCR and
do not reproduce that column. The text layer's parses come from a tree whose PDF, DOCX and
PPTX parsers are at versions 1, 2 and 2. Downstream runs against a Loom whose document
extractor is the stub; the decision counts add `--decisions 12` and the server's settings
for a real LLM (`python -m bench.docparse --help`):

```bash
PYTHONPATH=$PWD .venv/bin/python -m bench.docparse score text_layer=bench/results/docparse_parses/text_layer.jsonl layout_ocr=bench/results/docparse_parses/layout_ocr.jsonl baseline=bench/results/docparse_parses/baseline.jsonl --split heldout --json out.json
PYTHONPATH=$PWD .venv/bin/python -m bench.docparse run --out layout_ocr.jsonl --artifacts art/layout_ocr
PYTHONPATH=$PWD .venv/bin/python -m bench.docparse run --extractor <text-layer tree>/extractor --out text_layer.jsonl --artifacts art/text_layer
PYTHONPATH=$PWD .venv/bin/python -m bench.docparse artifacts bench/results/docparse_parses/baseline.jsonl --out art/baseline
PYTHONPATH=$PWD .venv/bin/python -m bench.docparse downstream text_layer=art/text_layer layout_ocr=art/layout_ocr baseline=art/baseline --base-url <loom> --token <the stub's token> --out downstream.json
```

### Memory at scale

`bench/scale_load.py` loads synthetic memories straight into the stores and then runs the
memory ops through one server built from source. Every key has 1 to 4 versions (2.4 on
average), `created_at` spreads over 360 days, 13 monthly partitions, and each embedding
row carries two 3072-dimension vectors (the fake embedder's width, which every `Quantized`
column is declared for). Two shapes:

- **single**: one org, one namespace, one agent and user `-`, so every search and every
  conflict check reads the whole namespace;
- **zipf**: 10,000 namespaces over 1,000 orgs, sized by a Zipf law of exponent 1, with 4
  agents and 100 users per namespace; a search reads one agent's and one user's rows.

The runs used local containers on a shared host: one machine with 32 cores and 61 GB,
ClickHouse 26.10 capped at 12 GiB (`max_server_memory_usage`), Postgres 16 in Docker with
`shared_buffers` 2 GB and no standby, other workloads beside them (1-minute load 13 to
224, recorded with every op). The Postgres latencies are pessimistic: a managed Postgres
on local NVMe commits faster, though a synchronous standby adds a round trip. The host's
free disk bounded the sizes; 1M keys and up did not fit beside both stores. Compare rows
of one setup, not absolute times. Every figure below is in
[`bench/results/scale_memory.json`](https://github.com/ClickHouse/loom/blob/main/bench/results/scale_memory.json), each run with
its setup and, where the harness recorded it, its build.

**Bytes per key**, with the row shape these runs wrote (a commit's full `loom.memory` row
as JSON in Postgres), from the main runs (single 250k and 500k, zipf 200k), each
the first size its run recorded; later snapshots in the artifact and the 20,000-key runs
carry somewhat more per row, from parts with few rows. The "Per key" column is the single
250k Postgres run's bytes of each table after its ops divided by its 252,103 keys:

| Store      | Table               | Bytes                                           | Per key |
| ---------- | ------------------- | ----------------------------------------------- | ------: |
| ClickHouse | `memory_embeddings` | 13,170 to 13,212 per row                        |  13,212 |
| ClickHouse | `memory`            | 90 (single) to 163 (zipf) per head after merges |      92 |
| ClickHouse | `memory_versions`   | 107 to 110 per version                          |     264 |
| Postgres   | `memory_current`    | 1,594 to 1,623 per head                         |   1,623 |
| Postgres   | `memory_revisions`  | 2,149 to 2,150 per revision                     |   5,136 |

**Cost per million memories**, at list prices ([ClickHouse Cloud
pricing](https://clickhouse.com/pricing), AWS us-east-1). Storage: 13.6 GB in ClickHouse
is $0.34 a month at $25.30 per TB-month; the 6.8 GB in Postgres is on the instance's own
disk, which ClickHouse Managed Postgres prices per instance rather than per GB. Compute:
ClickHouse CPU per call sent, refused calls' statements included, measured on 20,000 keys
in one namespace with ClickHouse capped at 4 GiB and 256 MB of Postgres `shared_buffers`
(`clickhouse_cpu_ms_per_call`, each statement's `OSCPUVirtualTimeMicroseconds`), priced at
$0.29846 per unit-hour of 2 vCPU. It is a floor: it prices CPU as if the service ran fully
used, and leaves out merges and the time a service sits idle:

| Op                            | CPU per call (ms) | Per million calls |
| ----------------------------- | ----------------: | ----------------: |
| `memory.get`                  |           5 to 16 |    $0.22 to $0.68 |
| `memory.set`                  |               215 |              $8.9 |
| `memory.search`               |        108 to 182 |      $4.5 to $7.6 |
| `memory.bulk_set`, 20 entries |             1,761 |               $73 |

The read ranges span the two authorities' runs. The write rows are the ClickHouse
authority's: under Postgres a write's ClickHouse inserts are sent by the publication
batch, whose statements are not tagged with the op, so that run's write figures leave them
out. A `set` and a `search` scan their scope, so their CPU grows with it, as their bytes
do (below). Postgres's compute is the instance's and was not metered per call. In the
Postgres run ClickHouse stalled, inserts past their deadline and reads timing out: of 200
`set` calls, 10 were answered 504 `BACKEND_TIMEOUT`, 2 were 503 `POSTGRES_UNAVAILABLE` and
6 passed the harness's 120 s client timeout; 32 of 200 `search` calls were 503
`PROJECTION_BEHIND`; and all 20 `bulk_set` calls were refused (13 `PUBLICATION_PENDING`, 7
`PROJECTION_BEHIND`).

**Ops**, p50 / p99 in ms at concurrency 5, 400 calls each (40 for `memory.bulk_set`, 20
entries a call), with the calls refused:

| Setup        | Authority  | `memory.get` | `memory.set` | `memory.bulk_set` (batched read) | `memory.search` | Refused      |
| ------------ | ---------- | -----------: | -----------: | -------------------------------: | --------------: | ------------ |
| single, 250k | Postgres   |     29 / 140 |  406 / 7,282 |                  21,614 / 41,906 |     697 / 2,446 | 1 bulk (503) |
| single, 250k | ClickHouse |     38 / 273 |  464 / 1,520 |                  22,104 / 57,220 |     739 / 2,703 | none         |
| single, 500k | ClickHouse |  115 / 1,839 |  619 / 6,758 |                          not run |         not run | none         |
| zipf, 200k   | Postgres   |     93 / 559 |  454 / 4,861 |                    2,284 / 6,126 |     347 / 1,849 | 8 bulk (503) |
| zipf, 200k   | ClickHouse |    120 / 763 | 909 / 28,071 |                    2,016 / 4,022 |         not run | none         |

The single 500k run stopped after `memory.set`, when the disk reached its floor; the zipf
ClickHouse run stopped after `memory.bulk_set`, when ClickHouse stalled (a `SYSTEM FLUSH
LOGS` ran past 180 s), and started on 186,494 heads, after the Postgres run's erasures.
Each "Refused" figure counts calls answered 503; no call of these runs was answered 500.
Neither authority was faster on every op: Postgres had the lower `get` and `set` p50,
ClickHouse the lower `set` p99 at 250k, in single runs on this host. Under Postgres
`memory.set` made 3.4 new parts a second per memory table at 250k, against 9.9 under
ClickHouse.

**The vector leg** (the statement of `memory.search` that ranks the scope's vectors,
codes-first), searched one call at a time:

| Scope                                           |  Namespace rows | Parts | Read (MiB) | p50 (ms) |
| ----------------------------------------------- | --------------: | ----: | ---------: | -------: |
| single, 250k                                    | 277,628 or more |    59 |        185 |      312 |
| the same after `OPTIMIZE TABLE … FINAL`         |         277,628 |    14 |        165 |      160 |
| single, 500k                                    | 527,628 or more |    69 |        283 |      301 |
| zipf, the largest namespace, one agent and user |          15,166 |    54 |         52 |      117 |
| zipf, the third namespace, one agent and user   |           6,285 |    53 |         43 |       69 |

In the single shape the scope is the namespace, and the leg reads about 0.56 to 0.70 KB
per row of it (the probes recorded no row count: the 250k rows are those after the
`OPTIMIZE`, which only removes rows, and the 500k rows are the load's). After the
`OPTIMIZE`, `memory.search` p50 fell from 671 to 392 ms at 250k. In the zipf shape the
scope is one agent's and one user's rows, about 1/400 of the namespace's: a scope of a few
dozen rows still reads tens of MiB, a granule of every part of its partitions.

**The write path's conflict check** reads the writer's scope too. At 250k, each
`memory.set` read 164 to 185 MiB. With the batched read, `memory.bulk_set` read 6.0 to 6.6
GiB a call, held up to 4.7 GiB in one statement and failed 65 to 93 statements a round of
20 calls on the memory cap, leaving 5 to 22 entries unchecked; with
`LOOM_MEMORY_BULK_PEER_READ=per_fact` it read 3.0 to 4.1 GiB, held at most 154 MiB and
failed none (three rounds each, alternated on the same rows).

**Erasure** at zipf 200k (`erasure.request`, purge), of different orgs and users in each
authority's run:

| Authority  | Scope                  |   Keys | Seconds | Outcome                                       |
| ---------- | ---------------------- | -----: | ------: | --------------------------------------------- |
| Postgres   | one user               |    590 |     235 | completed                                     |
| Postgres   | an org's 10 namespaces | 14,125 |     184 | completed                                     |
| ClickHouse | one user               |    113 |     186 | 504 `BACKEND_TIMEOUT` (a write past 60 s)     |
| ClickHouse | an org's 10 namespaces | 52,071 |     236 | failed: `memory.tombstone` read past its 30 s |

The org erasures ran with `external: false`, which skips the external backends' steps; on
these stores those are unconfigured stubs that erase nothing. During the ClickHouse runs
the erasure's mutations of `memory_embeddings` held the server: a `count()` over 19,941
rows took 153 s and a `system.mutations` lookup 184 s.

**Postgres** at zipf 200k (200,000 heads, 480,000 revisions as loaded): `VACUUM (ANALYZE)`
took 0.49 s for `memory_current` and 0.56 s for `memory_revisions`. After the Postgres
run's erasures (186,491 live heads), with 52,122 dead heads and 120,482 dead revisions
(the erasures' deleted rows plus a fifth of the rest rewritten in place), it took 0.57 s
for `memory_current` (275 MB of heap as loaded, so at least 0.48 GB/s) and 0.89 s for
`memory_revisions` (969 MB as loaded, at least 1.09 GB/s), both tables in the page cache;
a `COPY` of the largest org's rows of every memory table took 8.9 s for 351 MB.

**Several serving processes.** `memory.set` was sent open loop (each write at its time
slot, whatever earlier writes were doing, up to 512 in flight) at 15 and 150 a second over
two replicas of two web workers each (Postgres authority, small data, ClickHouse capped at
4 GiB). At 15 a second the host completed 8.6 a second; at 150 the 512-write cap held what
was sent to 57 a second, and the host completed 24 a second. ClickHouse got 4.3 to 5.7 new
parts a second per memory table, no partition passed 55 active parts
(`parts_to_delay_insert` is 1,000) and at most 11 merges ran at once, so merges kept up.
The refusals were the host's limits, all 503s: `PROJECTION_BEHIND`, `ADMISSION_CONTENDED`
and `OVERLOADED`. A rate of 1,500 a second would only raise the writes not sent at the
cap, and was not run.

**Extrapolated, and the ceiling hit first.** Each figure grows linearly with what it
reads:

1. **A read scope's vector scan (no vector index)**, at about one million memories in one scope
   (org, namespace, agent, user). At 0.6 KB a row, a search and every write's conflict
   check read 0.6 GB at 1M rows, 6 GB at 10M and 60 GB at 100M, each call: seconds per
   call before 10M. The size of the largest scope sets it, not the total number of
   memories.
2. **Erasure under the ClickHouse authority** is an observation, not a measured
   ceiling: at 200k keys, on a ClickHouse that had already stalled, its mutations of the
   vector table held the server's reads past their deadlines.
3. **One Postgres primary**, at 100M to 1B keys: 6.8 KB a key is 680 GB at 100M and
   6.8 TB at 1B. At the rates above, from tables in the page cache, a vacuum pass over 1B
   heads (1.6 TB) is near an hour and over their revisions (5.1 TB) over an hour, longer
   from disk; a one-org copy at 39 MB/s is half an hour for an org of 10M keys.
4. **ClickHouse storage**: 13.6 KB a key is 1.4 TB at 100M and 13.6 TB at 1B, nearly
   all of it vectors.

Reproduce, on a private ClickHouse and Postgres, with a server from source with background
workers off, auth off and the fake embedder (`.env` for the ClickHouse authority,
`.env.pg`, a copy of `.env` that adds `LOOM_STORAGE_AUTHORITY=postgres` and
`LOOM_PG_DSN`). The artifact records each run's setup (shape, size, caps, setting values);
the commands follow this pattern:

```bash
B="env PYTHONPATH=$PWD .venv/bin/python bench/scale_load.py"
$B load-ch --shape single --rows 250000     # or --shape zipf --rows 200000 --tenants 10000 \
                                            #   --orgs 1000 --agents 4 --users 100
$B --env .env.pg load-pg                    # into empty Postgres tables
$B --env .env.pg authority --to postgres    # start a server of that authority, then:
$B --env .env.pg pg --dump-org <org> --out <run>-pg-loaded.json      # vacuum and dump the rows as loaded
$B --env .env.pg measure --base-url <server> --shape single --out <run>-measure.json
$B --env .env.pg erase --base-url <server> --user <org>/<user> --org <org> --out <run>-erase.json
$B --env .env.pg pg --dead-fraction 0.2 --out <run>-pg-dead.json   # then vacuum with dead rows
$B --env .env.pg optimize --base-url <server> --shape single --out <run>-optimize.json
$B --env .env.pg fleet --base-url <server>,<server> --rate 15 --seconds 40 --out <run>-fleet.json
$B authority --to clickhouse                # then measure and erase without --env .env.pg
```

`authority --to` switches the recorded authority between runs on these private stores,
which a deployment never does. `measure --ops` and `--calls` narrow a run (the bulk peer
read rounds ran `--ops memory.bulk_set,memory.set --calls 100`, alternating the server's
`LOOM_MEMORY_BULK_PEER_READ`), `pg` without `--dead-fraction` times a vacuum of the rows
as loaded, and `load-ch --offset` grows a load (the 500k run grew the 250k one).

### A cell's org triggers

`bench/cell_trigger_cost.py` measures what a [cell](/architecture/cells)'s org
triggers add to writes of a guarded table (`loom.scope_generations`), in a database of
its own on a local Postgres 16 in Docker on a shared host, with the cell in `enforced`
and the written org held. It needs no provider and no server:

```bash
PYTHONPATH=$PWD LOOM_PG_DSN=postgresql://... .venv/bin/python bench/cell_trigger_cost.py
```

| Write                                 | Triggers off      | Triggers on       |
| ------------------------------------- | ----------------- | ----------------- |
| marked single-row insert, p50 / p99   | 3.08 ms / 21.0 ms | 2.38 ms / 14.1 ms |
| unmarked single-row insert, p50 / p99 | 3.26 ms / 19.1 ms | 2.56 ms / 11.7 ms |

| 500,000-row `COPY`, on minus off within a round | Median        | Over 6 rounds    |
| ----------------------------------------------- | ------------- | ---------------- |
| marked session                                  | 0.16 µs a row | −0.36 to 0.49 µs |
| unmarked session                                | 13.4 µs a row | 12.1 to 13.9 µs  |

Each single-row figure is the median of 5 runs of 2,000 inserts, the runs alternating
triggers off and on; the harness checks both triggers are on, at origin, in every run
that says so (after a bulk load an autovacuum can hold the table's lock past `ensure`'s
100 ms, and the harness then tries again, as a later boot would). The commit and a busy
host dominate the single-row figures, so they show no measurable cost, not a gain. Each
`COPY` round times both sessions with the triggers off and on, in an order that rotates,
and the cost is the difference within the round. A marked session, which every Loom
process with `LOOM_CELL` set opens, shows no cost the rounds can tell from zero (the six
rounds' differences range from −0.36 to 0.49 µs a row); an unmarked one pays 12 to 14 µs
a row for the state read and the `loom.tenancy` lookup. Loom's own writes are marked, so
this predicts the ship gate (a `memory.set` p99 rise of at most 10 ms, a delivery pass
at most 10% slower) holds by a wide margin; the cost lands on scripts and older builds
writing a cell without the check. This measures the triggers on the statements, not a
whole `memory.set` through a server on a cell; the end-to-end gate follows.

A small local diagnostic on an attached first cell records 400 sequential `memory.set`
calls per run and 1,000-row statement inserts from marked and unmarked sessions, three
rounds with org triggers on and off. Its fixture has two server processes, a router,
Postgres 16 with a synchronous standby and ClickHouse in Docker on a shared host under
other load. These are diagnostic figures, not the end-to-end acceptance workload:

| Local, three rounds                 | Triggers on            | Triggers off           |
| ----------------------------------- | ---------------------- | ---------------------- |
| `memory.set` p50                    | 154.4, 99.3, 98.4 ms   | 91.9, 101.2, 101.6 ms  |
| `memory.set` p99                    | 494.0, 291.7, 257.6 ms | 279.2, 244.7, 233.3 ms |
| 1,000-row insert, marked (median)   | 11.0, 9.2, 8.5 ms      | 9.7, 11.1, 10.6 ms     |
| 1,000-row insert, unmarked (median) | 22.1, 21.4, 21.0 ms    | 10.9, 9.6, 11.0 ms     |

The marked statement medians average 9.6 ms on and 10.5 ms off. A statement insert omits
the delivery worker's claim, embedding, publication and settlement work, so these figures
establish no delivery-throughput gate. The API p99 differences within each round are
+214.8, +47.0 and +24.3 ms; another small diagnostic has differences of −71.7, −10.0 and
−42.4 ms. Their opposing signs on the loaded host leave the API gate unresolved. Neither
end-to-end gate is established by these measurements.

`bench/cell_trigger_gate.py` measures the acceptance workload on an independently owned,
attached scratch fixture with explicit fake providers, enforced mode and an already held
org. Its namespace uses `append_only` for independent writes: sibling values do not
rewrite one another's heads. The fixture verifies the stored policy before every arm,
and the row proof requires each exact returned version and active status. This measures
trigger cost for independent writes, not semantic conflict resolution. Each API run has
50 unmeasured warmups and 2,000 sequential measured requests with
4,096-byte values, with five runs per arm in counterbalanced order. Each delivery block
times five complete `pg_delivery.run_pass` calls at the unchanged configured batch size,
with five blocks per arm. The latency gate is `median(p99_on) - median(p99_off) <= 10 ms`;
the throughput gate is `1 - median(rows_per_second_on) / median(rows_per_second_off) <= 0.10`.
The driver retains every run and raw outcome; it does not substitute statement timings.

```bash
PYTHONPATH=$PWD .venv/bin/python bench/cell_trigger_gate.py \
  --base-url <loopback-server-url> --pg-dsn <cell-dsn> --token <writer-token> \
  --org <held-org> --namespace <namespace> --fixture-manifest <fixture-manifest.json> \
  --launch-manifest <launch-manifest.json> --output <result.jsonl>
```

The fixture owner supplies the exact source, launch, configuration and ownership receipts,
excludes competing controllers and workers, and stops its processes before deleting its
exact scratch databases. The output path must not exist. The driver verifies source,
launch and configuration identities and the live fixture before native setup and every
arm, restores the org triggers within its cleanup bound and emits a final verdict only
after cleanup and artifact sync. It records the ownership-receipt digest supplied by
the fixture owner.
Source tests and the diagnostic figures above are not evidence that this workload passes.

## Reference computations

Deterministic outputs of the scoring code on fixed input, pinned by unit tests. The
methodology rests on them; they do not measure Loom.

### Lexical scoring reference values

`bench/answer_scoring.py` reproduces the published lexical tokenisation. A unit test pins
its output on one item (gold `7 May 2023`):

| Prediction                                                 | Token F1 | BLEU-1 |
| ---------------------------------------------------------- | -------- | ------ |
| `Caroline went to the LGBTQ support group on May 7, 2023.` | 0.3750   | 0.2308 |
| `May 7, 2023` (simplified)                                 | 0.8571   | 0.7500 |
| `8 May 2023` (wrong)                                       | 0.6667   | 0.6667 |

The wrong answer outscores the correct sentence, so lexical columns are computed only
after simplification and never replace the judge.

```bash
.venv/bin/python -m pytest tests/unit/test_answer_scoring.py -q
.venv/bin/python -c "from bench.answer_scoring import token_f1, bleu1
for p in ['Caroline went to the LGBTQ support group on May 7, 2023.', 'May 7, 2023', '8 May 2023']:
    print(round(token_f1(p, '7 May 2023'), 4), round(bleu1(p, '7 May 2023'), 4), p)"
```

### Bootstrap intervals on fixed deltas

The interval widths behind the noise floor: see
[methodology](/benchmarks/methodology#how-wide-the-interval-is) for the table and command.

### Majority voting on simulated reader noise

`test_majority_voting_narrows_the_null_band_on_synthetic_reader_noise` in
`tests/unit/test_bench_gate.py` simulates a reader that flips each label with a fixed
probability on 400 items. It asserts that a three-repeat majority narrows a
same-configuration band and more than halves between-arm discordance. This is why gates
use `--repeats 3`.

```bash
.venv/bin/python -m pytest tests/unit/test_bench_gate.py tests/unit/test_bench_bootstrap.py -q
```

## Scorer self-checks

Each runs offline on committed fixtures and exits non-zero when the scorer miscounts. The
printed scores are the ceiling of a perfect synthetic retrieval, not Loom's.

| Command                                                       | Asserts                                                                                                   |
| ------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| `make bench-code-intel-dry`                                   | Every lane of `code_intel_qa.jsonl` scores its ceiling under a perfect retrieval                          |
| `.venv/bin/python bench/coding_agent_qa.py --dry`             | R@5, R@10, MRR and fix-reuse compute on the committed lesson set                                          |
| `.venv/bin/python bench/code_plane_ablation.py selftest`      | The ablation scorer gives the expected scores on the arm A / arm C transcripts                            |
| `.venv/bin/python bench/acceptance/code_agent.py --dry`       | The acceptance report passes on synthetic retrievals                                                      |
| `.venv/bin/python -m pytest bench -q`                         | Every harness's scoring core and dataset reader (also run by `make test-unit` and CI)                     |
| `.venv/bin/python -m pytest tests/unit/test_skill_eval.py -q` | The bad replay fails with the expected finding codes, the good one passes ([skill eval](/benchmarks/skill-eval)) |

## What is not committed

- **Gate thresholds are not results.** The 0.80 quality floor and the `bench-code-intel`
  floors in the `Makefile` say what a build must clear, not what any build scored.
- **Fixture headers record gate history.** Some JSONL headers (for example
  `code_intel_qa_multilang.jsonl`) state the baseline a gate was accepted at. Reproducing
  it needs a live index of the same repository, so this page does not restate it.
- **Harness docstrings mention measurements** (the null band, flip rates, acceptance
  floors). They explain a default; they are not reproducible from committed artifacts. The
  code's defaults (`DEFAULT_BAND_PP`, `GATE_REPEATS`) are the operative part.
- **Per-run outputs** stay local ([where results land](/benchmarks/running#where-results-land)).
- **Consumer replays** need data outside this repository.

## Adding a result

Commit the artifact the harness writes, named after the benchmark, with the models it ran
under (`bench/models_used.py`: embedder and server LLMs, plus the bench's reader and judge)
and a reproduction command. Add both a row to the
[protocol index](#published-benchmark-protocol-index) and a subsection under
[measurements](#measurements). The index row must state:

- the exact task presented to Loom, including anything the evaluator withholds;
- the public corpus, selection method, random seed and number of questions or operations;
- the Loom operation, indexed scope, search mode and retrieval depth;
- the gold label, scoring rule, reader, judge and baseline;
- whether the run measures retrieval, answer quality, an agent outcome or efficiency.

The detailed subsection must link the per-example artifact, model manifest and exact
command, and must report failures, exclusions, uncertainty and material negative results.
A number without this evidence does not go here or in the README.

## Related

- [Running](/benchmarks/running) and [methodology](/benchmarks/methodology).
- [Code index](/features/code-index/overview): where `code.ask` and its cache fit.
