How much of your AI flow can run unreviewed?
Two steps. First, how much your labeled history can carry under an error cap. Then, what that frees in reviewer hours and money.
How much can your data carry?
The share of decisions that can go out without review while errors among them stay under your cap, and how many labeled cases that takes. Estimate it from three numbers, or run it on your own labeled history.
At most this share of errors among what goes out unreviewed, with 95% confidence.
Share of all decisions that turn out wrong, in percent.
How well the model's confidence separates right from wrong. 0.5 is chance, 1.0 is perfect. Measure it per case type: an AUC pooled over several kinds of cases can make this estimate too high.
Decided cases where you know whether the AI was right or wrong.
In percent, for example agents correcting decisions or QA sampling.
With 2,500 labeled cases, about 35% of decisions can likely be released, with errors among them proven under the 5% cap (95% confidence).
The signal tops out at about 55% of decisions at 5%, however many labeled cases you have. 50% takes about 24,000 labeled cases, about 11 months at your pace.
Planning estimate: at that many labeled cases, certification succeeds in about four runs out of five if the error profile holds.
| Error cap | With your labeled cases | With more labeled cases | Labeled cases needed |
|---|---|---|---|
| 1% | 0% | 0% | – |
| 2% | 0% | 15% | ~67,000 |
| 5% | 35% | 50% | ~24,000 |
| 10% | 75% | 85% | ~45,000 |
Show the curve as a table
| Labeled cases | Share releasable |
|---|---|
| ~1,800 | 30% |
| ~2,300 | 35% |
| ~3,500 | 40% |
| ~6,600 | 45% |
| ~24,000 | 50% |
What does that free?
Reviewer hours and money freed per month, from your case volume, review effort, the audits that continue and a share released without review: your own assumption, or step 1's estimate.
A random sample of released cases is still reviewed, so the watch can see errors. That time is not freed.
How the estimate works
Step 1 estimates, at 95% confidence, how much your labeled history can carry, from the numbers you enter. The Estimate tab uses a standard model of how scores spread for right and wrong decisions at the AUC you give; with your own data, your actual error profile is used. Nothing you enter leaves your browser.
The curve is a plan, not a guarantee: at that many labeled cases, about four runs in five succeed if the error profile holds.[F5] In live use the gateway keeps checking audited cases and the score mix, and hands decisions back to people when either moves past its alarm line.
The share in this calculator is an estimate or your own assumption, not a measured or certified figure. What our method proves is a limit on the error rate among the cases released without review, at your cap. Your own number is measured in a pilot. How a pilot works →
Your own number
Fifteen minutes of demo, then a pilot on your own flow.