NumGate
Every number in LLM-written sales notes, checked against its table
Run

When the Checker Is Noisier Than the Model

NumGate checks every number in LLM-written weekly sales notes against the table it came from. Each number is matched to a table cell, derived from the table, skipped, or flagged.

Results for this run

Filters

What the filters mean
  • Every chip starts on, so all notes are shown. Press a chip to leave that group out; press it again to bring it back. One chip in each group always stays on.
  • S1 Direct: the table is in the prompt.
  • S2 Compute-then-write: the model writes pandas code, the code runs, and the note is written from its results.
  • S3 Write-then-verify: an S1 draft is checked by the gate and corrected for up to two rounds.
  • S4 Facts-template: numbers are copied verbatim from a precomputed facts packet (an upper bound).
  • Rows are: whether each table lists countries or products.
  • Filters apply to 01 to 04 and to the gate bar at the bottom of the screen. They do not change the key figures at the top, 05 or 06, except the filtered-notes button in 05 (c).
01

Gate flag rates

For the filtered notes. 95% intervals from resampling notes.
Rates and charts use gate v3, the current gate. Gate v1 was kept only as a flag on each number, so its claim-level rates cannot be recomputed here.
02

The sorting gate

One dot per number. Hover or tap a dot to read its sentence.

matches a table cellderived from the tableskipped flagged by the current gateflagged by gate v1 onlycleared on table check (stet)
Sorting gate counts as a table
03

Rates by group

Where gate v3 flags cluster: by strategy, kind of claim, table size, cost and revision round. These are flag rates, not confirmed errors. Hover or tap a mark to highlight the same group everywhere.
03.1

Flag rate by strategy

Flagged: not found in table

Flagged: wrong entity, metric or period

03.2

Kinds of claim flag

03.3

Flag rate by table size

03.4

Not-found flags vs cost

03.5

Write-then-verify, round by round

03.6

Rates by group, as a table

04

All notes

Click a note to open it with every number marked.

Background: trust and method

05

How far to trust a flag

Filters affect only the filtered-notes button in (c)(a) and (b) are fixed results from the paper. (c) starts at the chosen run's rate.

A flag is a lead, not a verdict. When real errors are rare, even a small false-alarm rate means most flags are false alarms. (a) shows the Opus pilot under three checkers, (b) the measured accuracy on test sets, and (c) works out how many flags to expect to be real.

(a) Same notes, three checkers (Opus pilot)

(b) Accuracy on test sets and the pilot

(c) What a flag is worth

How the model works

precision: share of flags that are real; recall: share of wrong values caught on fresh test set 2; fa: false-alarm rate on correct numbers; b: share of numbers wrong (1 in N).

Flags are leads, not verdicts: check each one against the table.

06

Paper and method

Preprint

If nothing downloads, the same file is paper/numgate.pdf in the project.

Method notes
  • Level 1 checks each number against the table. Level 2 checks each claim's entity, metric and period.
  • Each rate is total flagged over total checked, across all filtered notes.
  • A flag is the gate's call. A confirmed error is a flag the blind AI review marked wrong (full study) or the table check found wrong (Opus pilot).
  • Intervals are 95% percentile bootstrap intervals over notes (1,000 resamples), computed in the browser, so they can differ slightly from results/summary.csv.
  • Each table, strategy and model was run twice (repeat 1 and 2).
  • S2 notes whose fact-computing code failed twice (no note written) are left out of rates.
  • Opus pilot only: gate v1 flags come from re-scoring the pilot notes with the first version of the gate (paper/rescore_v1.py).
  • The table check is the paper's check of every pilot flag against the source table, with AI assistance and programmatic cell lookups, not a human audit. The full study's verdicts come from a blind review by AI agents (paper/adjudication).
  • Rebuild this page with python src/dashboard.py (src/analyze.py also runs it).