Describe the job once, with the question, any pictures or files that go with it, and the right answers if you have them. Every model gets the same inputs. The report lays the answers out next to each other with the real price of each call, and if you gave it the right answers, it scores each model too.
model-bakeoff
model-bakeoff asks 19 different AI models the same question at once, then shows you every answer side by side and what each one cost.
Private buildCLI
$ python3 bench.py --job job.json --out bakeoff-out Models: Gemini 3.1 Flash-Lite, Gemini 3.6 Flash, Gemini 3.1 Pro (preview) [n4.txt] -> Gemini 3.1 Flash-Lite ... ok -> Gemini 3.6 Flash ... ok -> Gemini 3.1 Pro (preview) ... ok [n8.txt] -> Gemini 3.1 Flash-Lite ... ok -> Gemini 3.6 Flash ... ok -> Gemini 3.1 Pro (preview) ... ok [n9.txt] -> Gemini 3.1 Flash-Lite ... ok -> Gemini 3.6 Flash ... ok -> Gemini 3.1 Pro (preview) ... ok ===== SUMMARY (avg per item) ===== model $/item $/1000 sec ok score Gemini 3.1 Flash-Lite 0.0002 0.17 1.3 3/3 93% Gemini 3.6 Flash 0.0004 0.38 5.0 3/3 100% Gemini 3.1 Pro (preview) 0.0011 1.13 8.8 3/3 100% Wrote out4/report.html and results.json
Measurements
model-bakeoff run, 4 Nov 2026, this machine. 3 notes x 3 models, measured tokens at list price. Score is field-level accuracy against hand-checked labels.
Table
| Model | $ / 1,000 calls | Score % | Seconds / call |
|---|---|---|---|
| Gemini 3.1 Flash-Lite | 0.173 | 93 | 1.3 |
| Gemini 3.6 Flash | 0.384 | 100 | 5 |
| Gemini 3.1 Pro (preview) | 1.13 | 100 | 8.8 |
What it does
Hard problem we solved
Keeping eleven secret keys out of sight
Eleven providers means eleven API keys, and a runner that an AI assistant drives is exactly the kind of place a key leaks, into a log, a command line, or a chat transcript.
So key handling is a hard rule. The runner reads each key at run time from the Keychain, the environment, or a private file, and it never prints one, stores one, or passes one on a command line. Claude itself never reads them. It writes the job file and starts the run. A provider with no key is skipped and the rest still run, and one failing provider does not stop the run.
The rest is kept small on purpose. Anthropic and Google use their own API formats and the other nine providers use the OpenAI-compatible one. Models, prices, and structured-output modes live in one JSON roster, so adding a provider is one row, and a model-not-found error is fixed by editing one row, not code. Cost is measured from the token usage each API reports times the listed price, so a wrong price changes the dollar column but not the ranking.
Where it stands
In use as a personal Claude skill, about 300 lines of Python plus the roster.
A demo iPhone app that puts a fictional fund's numbers in its partners' pockets.
Next in Fund systems