A 96% Accurate Model That Misses One In Five Referrals
Engineering proposes a model that suggests which support tickets need a specialist. On 1,000 labeled tickets it is 96% accurate and still misses 20 of the 100 that needed escalation. A rule that never flagged anything would score 90%.
The guide follows a PM through that decision. You count the four outcomes, calculate precision, recall and F1, then move the cutoff to see what each extra catch costs. Moving it from 0.50 to 0.30 catches 15 more required referrals and adds an estimated 135 minutes of review per 1,000 tickets. The team picks 0.30 even though F1 is lower, because 0.50 misses more referrals than the support lead agreed to accept.
The company, tickets and results are invented for teaching. Replace them with your own labeled examples before you present a decision.
Inside The Guide
- Precision, recall and F1 worked out on 1,000 labeled tickets
- A threshold comparison that turns extra flags into review minutes
- A test on 1,000 unseen tickets against the existing keyword rule
- A pilot recommendation with owners and stop conditions
- Spreadsheet formulas and the edge cases where F1 is undefined