Your invoice-extraction system processes 50,000 invoices monthly. Roughly 90% come back with high confidence scores. The team wants to estimate the true error rate among high-confidence extractions before deciding whether to reduce human review on them. Reviewing all 45,000 manually would take months.
What's the right measurement approach?
Why did you pick that answer? Two or three sentences. The act of articulating it is what builds the judgment — not the click that follows.
Stratified sampling reveals concentration risk that aggregate metrics hide. A simple random sample tells you the error rate; a stratified sample tells you whether errors are concentrated on photos vs. PDFs, on small vs. large invoices, on one-off vendors vs. recurring ones. Aggregate accuracy can mask catastrophic concentration in a specific subset (e.g., the largest-dollar invoices). Sampling lets you act on a small fraction of the data while still surfacing where review is most needed.
Reviewing all 45,000 defeats the entire point of confidence-based routing. The infrastructure exists precisely because exhaustive review doesn't scale.
The whole reason to measure error rates is that confidence scores aren't reliably calibrated. Trusting them without measurement is exactly the failure mode this question is meant to prevent.
The most recent 100 may not represent the population — vendor mix shifts, document types change, formats evolve. A non-random sample produces a non-representative estimate.