Switchcraft: AI Model Router for Agentic Tool Calling

We find two counter-intuitive results: larger models do not consistently outperform smaller ones on tool-use tasks, and nominally cheaper models can incur higher total cost due to token-intensive output. It comes from a paper posted to arXiv on 3 November 2026. Agentic AI systems that invoke external tools are powerful but costly, leading developers to default to large models and overspend inference budgets. Existing model routers target chat responses or end-task success, not the correctness of the individual tool calls on which agent tasks depend; we present Switchcraft, the first (to the best of our knowledge) model router optimized for strict tool-call correctness in agentic workloads. We construct an evaluation framework on six function-calling benchmarks, holding one out entirely, and fine-tune a calibrated DistilBERT-based classifier, deployed inline under a latency budget, to select the lowest-cost model subject to correctness or abstain when none is likely to answer correctly. Switchcraft reaches 78.3% answer accuracy (86.4% when correct abstentions are credited) at 59% lower cost than the best individual model (75.6%), saving over $3,800 per million queries, and outperforms kNN, MIRT, heuristic, and random routers. On the held-out benchmark, it matches or exceeds every individual model at 5--7X lower cost than the strongest ones.

Why it counts

We find two counter-intuitive results: larger models do not consistently outperform smaller ones on tool-use tasks, and nominally cheaper models can incur higher total cost due to token-intensive output. Switchcraft reaches 78.3% answer accuracy (86.4% when correct abstentions are credited) at 59% lower cost than the best individual model (75.6%), saving over $3,800 per million queries, and outperforms kNN, MIRT, heuristic, and random routers. Existing model routers target chat responses or end-task success, not the correctness of the individual tool calls on which agent tasks depend; we present Switchcraft, the first (to the best of our knowledge) model router optimized for strict tool-call correctness in agentic workloads.

Sources

Primary source: primary source

What is not known

This brief does not claim independent replication. Claims that appear only on X and not in the primary source stay unknown.