What three AI models got wrong on 20 pricing pages

Getting an AI model to return JSON is one thing. Getting the price, currency and billing period right is harder. We tested a small version of a task used by Agentery, turning public software pricing pages into structured facts, with two open-weight models through JohnRouter alongside the closed model that runs the production pipeline today.

Qwen3 had the lowest recorded inference charge. The baseline produced the most completely correct records. Annual billing and missing information exposed the useful differences. These are preliminary results from 20 deliberately selected pages; the reference answers were prepared and checked by an AI and still need human review, and three pages are explicitly disputed.

The result

Each endpoint received the same 20 frozen page inputs twice. A record passed only if all five requested facts matched the reference: product name, website, free tier, cheapest eligible subscription (amount, currency, unit) and billing period.

*USD. JohnRouter figures are the buyer charge returned with each response. The baseline figure is the charge OpenRouter reported to its own account per request (credits, excluding top-up fees). Different billing bases: these figures do not establish like-for-like customer savings. Provisional: reference answers pending human review; three pages disputed.

All 120 calls returned valid JSON with the required keys under the tested JSON-mode configuration, with no timeouts. That is useful, but formatting success did not guarantee factual correctness, and the test was sequential: it says nothing about uptime or behaviour under load.

The easy fields (product name, website) were right almost every time; they largely come from the page metadata. On the three harder fields together (free tier, subscription, billing period) the counts were 99, 94 and 86 of 120. Billing period was 33, 34 and 33 of 40: Qwen3 got one more right in this sample, a difference too small to support any conclusion.

Annual billing caused mistakes

One frozen page (100Hires) advertised its cheapest plan as €42/month, EUR, billed annually, next to a €86/month monthly option. The test rule asks for the amount actually charged for the billing period you return: €504.00 per year.

A monthly equivalent and an annual charge are both useful numbers, but they must be stored with the correct meaning and commitment. Two more pages went the same way: a chat tool at €6 a month or €60 a year, and a hosting plan at 7 EUR a month or 70 EUR a year. The rule requires the eligible plan with the lowest monthly equivalent, so the yearly charge was expected; the open-weight models returned the monthly plan on all four attempts each, the baseline the yearly charge every time. These are results against a particular extraction rule, not a judgement that annual billing is better for a customer.

Missing information tested the rules

The instructions required models to leave unsupported information unknown. A page that never mentions a free tier is not evidence that there is none.

Against the provisional references, the baseline followed the abstention rule on 29 of 30 eligible field-attempts, Qwen3 on 24 and Maverick on 20. This needs care: those 30 are not 30 independent pages. Each page was tested twice and the subscription and billing-period fields are linked. One disputed page (0xArchive) accounts for half of the open-weight models' recorded abstention errors: it prints "$49/mo" without ever naming the currency. Both open-weight models returned 49.00 USD per month on every attempt. The amount and the period are on the page; the currency is inferred from the "$" sign, which the frozen rule forbids. The other half were a confident "no free tier" on three pages that say nothing about one, where the baseline answered "unknown".

A plausible wrong value passes a JSON schema check, so it needs its own validation downstream: evidence checks, currency validation and arithmetic can catch some of these, but not for free. We are reviewing the currency rule before drawing broader conclusions.

The open models also caught things the baseline missed

On a pay-per-call API page with free tools (APIbase.pro), both open-weight models matched the reference on both attempts; the baseline answered "no free tier" both times. On a security product with a free community tier (AgentInterdict), Qwen3 matched both times, Maverick once, the baseline neither time: it returned no subscription at all although the page lists a £79-a-month plan. Same provisional reference set, same caveat. The model with the best overall score still missed individual products, which is the main reason to test the pages and rules that matter to your own application.

What the costs mean

The itemised charges for the 120 scored calls sum to $0.0593. The pilot's ledger shows $0.0684 in total: the difference is four baseline calls that failed on a mis-pasted key at the first launch, were charged at their conservative reservation ($0.0092) and re-run. Every reservation and settlement is in the package.

Scaled figures are arithmetic from the recorded 40-call charges, not 1,000-call experiments. The bases differ (see above), so no savings percentage is claimed. Charges are what the run actually paid; catalogue rates were used only for budget reservations.

Dividing charges by correct records is a diagnostic, not a quote. In the experiment the reference answers identify mistakes afterwards; a production workflow has no such oracle and must pay to detect, correct or tolerate wrong outputs. To show the scale: on these unnormalised charges Qwen3 costs about $0.39 less per 1,000 calls than the baseline, while its complete-record failure rate is 12.5 percentage points higher, which at this mix means about 125 more failed records per 1,000 calls. That is an illustration from a small provisional sample, not a forecast. Even modest extra checking could outweigh a token-cost difference of that size. For this task we would examine the pricing errors and the validation effort before changing the production model.

How we ran it

Four pages from each of five groups: straightforward subscriptions, monthly-versus-annual billing, free plans and trials, missing or usage-based prices, and mixed pricing with distractors. Each page was frozen as text with line references from a saved render (server-side, Helsinki, no locale cookie). Two of the first picks were replaced before any model ran (a two-line page, a suspended service); the selection record says so.

One shared prompt (13 rules, JSON-only output), temperature 0, JSON mode, 1,000 output tokens, 90-second timeout, one call at a time in a seeded interleaved order. All three endpoints accepted the same request body; the JohnRouter calls were pinned to its bridge provider with fallback off and every response came back from that route and model. A deterministic scorer compared outputs with the references using predeclared aliases; no model judged another. Repeatability is reported two ways: pages where both attempts were completely correct (10, 10 and 7 of 20), and pages where the two attempts gave the same canonical five-field answer whether right or wrong (14, 18 and 17 of 20).

The tested JohnRouter paths use its bridge provider. The upstream operator is not identified publicly, so these results describe JohnRouter's buyer-facing route; they do not certify an independent seller's hardware, price or privacy practices. Response times mix model, provider, route and network and are not attributable to JohnRouter overhead or to a model's intrinsic speed.

Reference review. Seven of the twenty reference answers carry an open question and three of those are explicitly contested (F06 Datablist: which annual figure is the account charge; F17 0xArchive: the currency rule; F19 Aida: whether an "Always free" allowance is a free tier). The review sheet lists the source passage, the rule, the alternative reading and its effect on the counts before any disposition. Rescoring the saved responses with F17 read as 49.00 USD/month gives 24, 23 and 19 completely correct: the ordering between the baseline and Qwen3 depends on that single decision. Removing the three contested pages gives 23, 20 and 17 of 34. The rules were not changed; a later change of task would be a new version.

Technical table and harder-field chart

model	route	completely correct	field accuracy	harder fields (of 120)	abstention rule followed	unsupported assertions	valid JSON	both attempts correct	same answer twice	median ms	p95 ms (n=40)	timeouts
openai/gpt-5.6-luna	OpenRouter direct	26/40	89.5%	99	29/30	1	40/40	10/20	14/20	3444	6612	0
qwen/qwen3-235b-a22b-2507	JohnRouter bridge	21/40	86.0%	94	24/30	6	40/40	10/20	18/20	4624	9245	0
meta-llama/llama-4-maverick	JohnRouter bridge	17/40	83.0%	86	20/30	10	40/40	7/20	17/20	2372	8440	0

Reproduce it

The package contains the 20 frozen fixtures with SHA-256 hashes, the prompt, the reference answers with their review status and the disputed-case sheet, the endpoint configuration without secrets, every raw response, the spend ledger, the scorer and a manifest of hashes. From a fresh extract, node score.mjs --out=. regenerates every number and chart on this page from the saved responses with no API key and no spend; we verified that from a clean directory before publishing. The exact original snapshots, responses and ledger are preserved unchanged in the repository; the change note records what this revision altered.

Download the benchmark package (zip, v1.1)

Try a task that matters to you