An invoice extractor outputs a confidence score per field — {vendor: "Acme Corp" (0.89), date: "2024-05-04" (0.92), amount: 1287.42 (0.71), invoice_number: "INV-9938" (0.68)}. The team currently routes any extraction with any field below 0.85 to human review. About half the time, those routed extractions turn out to be entirely correct. Meanwhile, several auto-approved extractions (all fields above 0.85) have turned out to be wrong.
What's the right fix?
Why did you pick that answer? Two or three sentences. The act of articulating it is what builds the judgment — not the click that follows.
Confidence scores aren't equally calibrated across fields — models tend to be well-calibrated on some fields and overconfident on others. A global threshold treats all fields as if they had the same reliability, which produces both kinds of errors: high-confidence wrong answers slip through (overconfident fields) and high-confidence correct answers get over-escalated (well-calibrated fields where 0.85 is genuinely high). Calibrate per field against labeled data, then route by field-specific thresholds. This is one of the highest-leverage moves in extraction-system design.
Lowering to 0.70 reduces escalations but lets through more of the overconfident-but-wrong cases that were already a problem. You're trading one error type for the worse error type.
Raising to 0.95 catches more cases for review but also escalates many extractions that are correct. Reviewer capacity gets consumed on extractions that didn't need it. Per-field calibration uses reviewer time where it actually pays off.
Reviewing everything forfeits the entire benefit of confidence-based routing. The data is there to calibrate properly; throwing out the routing layer is over-correction.