Cutting AI Processing Costs 40% Without Sacrificing Accuracy

Frontier language models are remarkably capable. They’re also remarkably expensive at scale. When every user action triggers an API call, and every call includes the full data schema, all field definitions, all options, and the complete input — token counts compound fast. At production volume, AI processing becomes the single largest infrastructure line item.

The instinct is to look for a cheaper model. The better question is whether every request actually needs the full context. In most production systems, the majority of AI requests are straightforward — clear inputs with obvious mappings. It’s only a minority of requests where the full context changes the outcome.

This observation led us to a tiered architecture we call Planner-Validator: a lightweight first pass handles the easy cases cheaply, and a full-context second pass handles only the cases where it matters. (The business framing of the same result: what 40% lower AI costs looks like.)

The Planner receives a compressed context — only the most frequently used fields based on historical analytics, limited options per field, and the input. For straightforward cases, this compressed context is sufficient for high-confidence extraction. The majority of requests resolve at this tier without escalation.

When the Planner’s confidence drops below a threshold, the Validator activates with the complete context — every field, every option, full business rules. The Validator also activates on two additional triggers: when the Planner returns no results (giving the full model a second look), and when complex business logic transitions are available.

The result: roughly 60-70% of requests resolve at the cheap tier. Only the 30-40% that genuinely benefit from full context pay for it. Overall token reduction: approximately 40%.

A fixed confidence threshold is a starting point, not a solution. Different data fields have different difficulty levels. A dollar amount mentioned explicitly in conversation is easier to extract than a categorical classification that requires inference from context.

We implemented adaptive thresholds that adjust per-field based on user confirmation patterns. Fields with consistently high acceptance rates earn trust — their threshold drops, keeping more requests at the cheap tier. Fields with lower acceptance rates have their threshold raised, sending more requests to the full-context Validator until accuracy improves.

Beyond threshold adaptation, the system learns recurring patterns that eliminate AI calls entirely. When users consistently confirm the same mappings, those patterns graduate to local resolution rules. Future occurrences are handled instantly with zero API cost. This creates a compounding efficiency: the longer the system runs, the cheaper it gets — without sacrificing accuracy on novel inputs. When AI consumption is metered to users, the same tiering shapes pricing — see credits over subscriptions.