Where our numbers come from
Every result on this site has a source. Here they are, with the conditions that apply.
F1How many companies review everything
Global survey by McKinsey, The state of AI: How organizations are rewiring to capture value (published 12 March 2025; responses collected in July 2024): 27% of respondents whose organizations use gen AI say that employees review all content created by gen AI before it is used. Source: mckinsey.com.
F2What is proven, and under which conditions
The error rate among the cases of one case type released without review is at most the declared cap, with probability at least 95% over the reviewed history used for calibration. It rests on exact statistical tests at 95% confidence, testing release shares from the safest share upward. This assumes live traffic resembles the reviewed history: the same model, the same prompts and the same kind of cases. Even with correct execution, about one calibration in twenty can end above the cap. That rate counts every calibration, including those that release nothing, so a case type that only just carries its cap is more likely to be one of them. With several case types, the chance that at least one ends above its cap is higher, unless a stricter level is agreed. A case type whose history cannot carry the cap is not released; it stays with people. The share released is measured, never certified: it depends on the customer’s AI, cases and cap, and is measured in a pilot.
F3Demo on open data
Our gateway demo, run on 14 October 2026: customer-support ticket routing on open data, with a simplified stand-in routing model and a 5% cap. Of 1,500 tickets, 55.7% were released without review, with a true error rate of 21 in 835 = 2.5% (95% interval 1.6–3.8%). A simulated unannounced model update, which shifted the score mix, raised an alarm from the score watch after 236 tickets; until then, released tickets carried the updated model’s higher error rate. Recalibration on the updated model released only 10%. A second class, FAQ answers, could not carry the cap and stayed with people. These are demo numbers on open data, not customer or pilot results; the true error rates are known only because the open data carries the right answers.
F4The watch: false alarms and what it sees
The error watch counts errors in a random audit sample of released cases; the score watch compares the mix of risk scores with the calibration history and needs no labels. While the error rate is at or below the cap and the score mix is unchanged, false alarms come on average no more often than once per 10,000 audited cases (error watch) or once per 100,000 decisions (score watch). These are averages, not per-check probabilities. The score watch sees changes that move the scores, such as a model swap; a rise in errors that leaves the scores unchanged shows up only in the audit sample, so the audit rate sets how fast it is caught.
F5The calculator’s planning estimate
A share counts as reachable when certification is likely to succeed, in about four runs out of five, under a planning error curve: a standard model of how scores spread at the AUC you give, or a fit to your own labeled data. It is a planning estimate, not a measurement, and it can be too high when the AUC is pooled over several kinds of cases.
Questions about a number?
We're happy to walk through any of them.