Which open-weight models can extract prices accurately? A wider JohnRouter test

The first pilot compared two open-weight models with the closed model a production pipeline uses. This one takes every open-weight chat model in JohnRouter's catalogue on 28 September 2026, 155 of them, gives each the same 20 pricing pages once, then validates the five best on 20 fresh pages, twice each. The closed control ran alongside on the same pages.

Short version: the top open-weight models matched the closed control on this task. None beat it in screening under the scored rules; two of the three that tied or came within two pages cost less per call. In validation the five finalists and the control landed between 31 and 34 correct records of 40, a spread of three pages that this sample cannot rank. Most of the catalogue did much worse, and a large share of failures were formatting, not facts: without an API JSON mode, 19 models never returned a bare JSON object in 20 tries.

Screening: 155 models, 20 pages, once each

Each configuration received the same 20 frozen page texts once: 3,100 open-weight attempts plus 20 for the control. A record counts as completely correct only when all five facts match the reference: product name, website, free tier, cheapest eligible subscription (amount, currency, unit) and billing period.

The top of the table, all reasoning models: MiMo v2.6 Pro 18, MiMo v2.6 Flash 17, Kimi K3 17, then six models at 16 (DeepSeek V4.1 Flash, Qwen 3.8 Flash, Qwen 3.5 27B, Kimi K2 Thinking, Qwen 3.8 27B, Qwen 3.8 2.4T). Two of the nine cost less than the control for the same 20 calls: MiMo v2.6 Flash at $0.0100 and DeepSeek V4.1 Flash at $0.0127 against $0.0175.

Reasoning models did clearly better here. The 91 reasoning configurations had a median of 12 correct; the 64 non-reasoning ones a median of 4. The best non-reasoning model was Cohere Command A at 15. The price of reasoning is time: the median open-weight response took 9.5 seconds against 3.1 seconds for the control, and the 4,096-token budget cut off 224 attempts, 20 of 20 for one model.

Formatting decided a lot of scores

This time the API's JSON mode was off for everyone. The prompt asked for a single JSON object and nothing else, and a reply that put the object inside prose or a code fence counted as a format failure. Nothing was repaired.

Forty-seven models returned valid JSON on all 20 pages. Forty-nine did so on fewer than 10, and 19 never did. Eleven of those 19 wrapped a perfectly good object in a Markdown fence every single time, among them the Gemma 3 family, the Mistral Small and Ministral models, Phi-4 and DeepSeek Chat V3-0324. As a diagnostic only, not part of the scored protocol, we pulled the JSON out of the prose and re-checked it: 399 answers came back, 35 models changed score, Gemma 4 31B went from 3 to 18 and GLM 5.1 from 1 to 16. Even then no model beat the control. If your pipeline can strip fences, or if your endpoint offers a JSON mode, the picture for those models is very different from the scored table; if it cannot, they failed the job as posed.

Where the facts went wrong

Among answers that parsed, the easy fields were mostly right: product name 2,026 and website 2,043 of 3,100. The hard fields were where models separated: free tier 1,617, subscription 1,682, billing period 1,770. Across the parsed answers there were 770 wrong values for facts the page states, 322 assertions with no support in the text, and 100 cases where a model returned a price and period for a page whose currency is never stated, which the rules require to be left null.

The same page that caught the first pilot's open-weight models caught most of the catalogue. Upfling advertises a plan at €2.83 a month billed yearly; the actual recurring charge is €34.00 a year. Forty-one models returned €34.00 for a year; 53 returned €2.83, mostly as a monthly price. On 100Hires (€42 a month billed annually against €86 monthly) 44 models returned the €504.00 annual charge, and 43 returned €42.00.

Two pages were failed by almost everyone. A quote-only page and a page with no subscription at all both require free tier "unknown" because the text says nothing about one; 83 and 87 of about 110 parsed answers said "no". A page for a product still in beta was answered correctly by 9 models: the reference has free tier "unknown", and the annotator flagged that reading as a question. Under the alternative reading ("no"), the control drops to 17 and two open-weight models edge past it. Under the other flagged alternative (an association plan at €150 a year on one page), the control rises to 19 and nothing passes it. Both re-scorings are in the package.

Validation: five finalists on fresh pages, twice

The selection rule was written down before screening ran: the three highest counts, then the cheapest model within two pages of the best, then the fastest within that band, complete coverage only, control excluded. It picked MiMo v2.6 Pro, MiMo v2.6 Flash, Kimi K3, DeepSeek V4.1 Flash and Kimi K2 Thinking. Twenty fresh pages, chosen and annotated before the finalists were known, were sent twice to each finalist and to the control.

*USD. JohnRouter figures are the buyer charge returned with each response; the control figure is the charge OpenRouter reported to its own account per request. Different billing bases. "Same answer twice" counts pages where both attempts parsed and returned identical facts, right or wrong.

The spread is three pages of 40. The two MiMo models and DeepSeek finished ahead of the control by one to three records, and the two Kimi models tied it. On this sample that is not a ranking; it is six configurations that do the job about equally well on average, at charges that differ by a factor of forty. Temperature 0 did not mean determinism: the control gave different answers on six of the 20 pages, and DeepSeek V4.1 Flash matched itself on only eight.

The fresh pages found new ways to be wrong. On a developer billing page three of the six configurations, the control included, reported the website as a /pricing URL that appears nowhere in the text, so an otherwise correct record failed on the easiest field. Five answers annualised a $50 monthly plan into $510 a year. Two pages that never mention a free plan were answered "no" instead of "unknown" by most configurations, and one page whose only free offering is a set of tools rather than a plan split the models on the same field. Each of those references carries an open question for the human reviewer, and they are listed in the package.

What changed since the first pilot

• Scope. Two open-weight models became 155: every open-weight text-chat model in JohnRouter's catalogue snapshot of 28 September 2026 14:28 UTC, resolved to a public weights listing. Fifty-five image models were excluded. One model, undi95/remm-slerp-l2-13b, never answered: all 20 requests were refused upstream, so it is counted as attempted, not as scored.
• No JSON mode. The first pilot switched on the API's JSON mode. This time the prompt alone had to carry the format, because that is the common denominator across 155 endpoints and because it makes format failures visible instead of hidden.
• Same rules, corrected reference. Two of the first pilot's 20 references were replaced with comparable pages of the same type after their page text was found genuinely ambiguous; one gained a website alias. The 18 other pages are unchanged. The two first-pilot candidates were screened again under the new protocol: Qwen3 235B Instruct scored 14 of 20 (rank 18 of 155, valid JSON every time, $0.0058 for 20 calls) and Llama 4 Maverick 11 (rank 53).
• Budget. One ceiling of US$10 for everything. Total settled charges were US$9.45 across 3,487 calls, plus US$0.03 of reservation left on the ledger by a stopped run (next item). The ceiling was never raised and nothing was trimmed.
• Two scorer bugs, fixed before publication. The duplicate-key check counted the five field names inside the required evidence block, so on a dry run every valid answer scored zero; fixed with a top-level-only check. Then the currency comparison demanded an ISO code while the prompt explicitly allows "€", "£", "US$" and similar; 205 answers from 68 models were affected, and the control's own score was 11 of 20 before the fix and 18 after. Because the second fix changed the finalist in slot three (Kimi K3 in, Qwen 3.5 27B out), a validation run that had started under the old table was stopped after 21 calls and restarted; its calls and charges are kept in the ledgers. Both fixes have regression checks, and the raw responses were never edited.

How we ran it

Screening ran from 14:43 to 19:24 UTC with a four-minute retry pass; validation from 19:34 to 19:47 UTC. The control ran through OpenRouter, outside the open-weight count and on a different billing basis. Nothing here measures uptime, rate limits or behaviour under load: 65 upstream 429 responses and 38 "no eligible provider" refusals in screening are recorded, but four concurrent calls is not a load test.

Reproduce it

The package holds the frozen inputs, both reference sets with the reviewer's open questions, every raw response, both spend ledgers, the per-model CSVs, the charts, the harness and the change log. The scorer runs offline from the raw responses; the runner needs your own keys.

Download the v2 package (zip) First pilot →

Try a task that matters to you