# JohnRouter extraction pilot — report (2026-09-28)

Task: extract five facts (product name, website, ongoing free tier, cheapest eligible paid subscription with actual billed amount/currency/unit, billing period) from 20 frozen public pricing/product page texts. 2 independent attempts per fixture per endpoint = 40 scored attempts per endpoint. Reference answers: **AI-prepared and AI-checked (Claude, 2026-09-28); provisional until Ben reviews review.csv** — results are provisional until Ben reviews review.csv. Scorer scorer-v1-2026-09-28; no LLM judge.

## Comparison

| endpoint | route | model | complete | completely correct | field accuracy | free_tier | subscription | period | known-value acc. | correct abstention | unsupported assertions | schema ok | repeat consistency | median ms | p95 ms (exploratory) | timeouts | total cost | cost / correct |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| baseline-luna-openrouter | openrouter-direct | openai/gpt-5.6-luna | 40/40 | **26/40** | 89.5% | 34/40 | 32/40 | 33/40 | 88.2% (170) | 96.7% (30) | 1/30 (+20 other errors) | 40/40 | 10/20 | 3444 | 6612 | 0 | $0.0271 (upstream_usage_cost) | $0.0010 |
| candidate-a-johnrouter | johnrouter | qwen/qwen3-235b-a22b-2507 | 40/40 | **21/40** | 86.0% | 30/40 | 30/40 | 34/40 | 87.1% (170) | 80.0% (30) | 6/30 (+22 other errors) | 40/40 | 10/20 | 4624 | 9245 | 0 | $0.0114 (johnrouter_billed) | $0.0005 |
| candidate-b-johnrouter | johnrouter | meta-llama/llama-4-maverick | 40/40 | **17/40** | 83.0% | 24/40 | 29/40 | 33/40 | 85.9% (170) | 66.7% (30) | 10/30 (+24 other errors) | 40/40 | 7/20 | 2372 | 8440 | 0 | $0.0208 (johnrouter_billed) | $0.0012 |

Cost basis: `johnrouter_billed` = the actual buyer-facing charge returned by JohnRouter per request; `rate_card_from_usage` / `upstream_usage_cost` = OpenRouter-direct supplier cost (intermediary; not a JohnRouter buyer price). Failed attempts stay in every denominator and their known or reserved cost is included.

## Charts

![cost per correct](charts/cost-per-correct.svg)
![completely correct](charts/completely-correct.svg)
![field accuracy](charts/field-accuracy.svg)
![hard fields](charts/hard-fields.svg)
![latency](charts/latency.svg)

## Failure examples

Confident wrong price (a determinate subscription where the reference has a different value):
- **F11 (blitz_cloud, G3 free plans and trials) · candidate-b-johnrouter · rep 1** — got subscription {"amount":"7.00","currency":"EUR","unit":"per_account"} / period "month" / free_tier yes; expected {"amount":"70.00","currency":"EUR","unit":"per_account"} / "year" / yes. Wrong: cheapest_paid_subscription, billing_period.
- **F11 (blitz_cloud, G3 free plans and trials) · candidate-a-johnrouter · rep 1** — got subscription {"amount":"7.00","currency":"EUR","unit":"per_account"} / period "month" / free_tier yes; expected {"amount":"70.00","currency":"EUR","unit":"per_account"} / "year" / yes. Wrong: cheapest_paid_subscription, billing_period.
- **F18 (business_studio, G5 mixed pricing and distractors) · baseline-luna-openrouter · rep 1** — got subscription {"amount":"150.00","currency":"EUR","unit":"per_account"} / period "year" / free_tier yes; expected {"amount":"25.00","currency":"EUR","unit":"per_account"} / "month" / yes. Wrong: cheapest_paid_subscription, billing_period.
- **F01 (100hires_mcp_server, G1 straightforward subscription) · candidate-a-johnrouter · rep 1** — got subscription {"amount":"42.00","currency":"EUR","unit":"per_account"} / period "year" / free_tier yes; expected {"amount":"504.00","currency":"EUR","unit":"per_account"} / "year" / no. Wrong: free_tier, cheapest_paid_subscription.

Unsupported assertions (a determinate answer where the reference requires unknown/null):
- **F17 (0xarchive_mcp, G5 mixed pricing and distractors) · candidate-a-johnrouter · rep 1** — got subscription {"amount":"49.00","currency":"USD","unit":"per_account"} / period "month" / free_tier yes; expected null / null / yes. Wrong: cheapest_paid_subscription, billing_period.
- **F17 (0xarchive_mcp, G5 mixed pricing and distractors) · candidate-b-johnrouter · rep 1** — got subscription {"amount":"49.00","currency":"USD","unit":"per_account"} / period "month" / free_tier yes; expected null / null / yes. Wrong: cheapest_paid_subscription, billing_period.
- **F20 (agent_utilities_2, G5 mixed pricing and distractors) · candidate-b-johnrouter · rep 1** — got subscription null / period null / free_tier no; expected null / null / unknown. Wrong: free_tier.
- **F07 (anjin_digital, G2 monthly vs annual) · candidate-b-johnrouter · rep 1** — got subscription {"amount":"149.00","currency":"GBP","unit":"per_account"} / period "month" / free_tier no; expected {"amount":"149.00","currency":"GBP","unit":"per_account"} / "month" / unknown. Wrong: free_tier.

Correct abstention (null subscription and period where the evidence has no eligible price):
- F15 (12port) · candidate-b-johnrouter · rep 1: returned null/null as required
- F15 (12port) · candidate-a-johnrouter · rep 1: returned null/null as required
- F15 (12port) · baseline-luna-openrouter · rep 1: returned null/null as required

Schema/format failures by endpoint:
- baseline-luna-openrouter: none
- candidate-a-johnrouter: none
- candidate-b-johnrouter: none

## Per-fixture results (completely correct attempts per endpoint, of 2)

| fixture | product | group | baseline-luna-openrouter | candidate-a-johnrouter | candidate-b-johnrouter |
|---|---|---|---|---|---|
| F01 | 100hires_mcp_server | G1 straightforward subscription | 1 | 0 | 0 |
| F02 | accountsos_mcp_server | G1 straightforward subscription | 2 | 2 | 1 |
| F03 | adcrunch | G1 straightforward subscription | 1 | 2 | 2 |
| F04 | adtest_ai | G1 straightforward subscription | 1 | 0 | 1 |
| F05 | agently | G2 monthly vs annual | 2 | 2 | 2 |
| F06 | datablist_ai_agent | G2 monthly vs annual | 1 | 0 | 0 |
| F07 | anjin_digital | G2 monthly vs annual | 1 | 2 | 0 |
| F08 | discordmcp | G2 monthly vs annual | 2 | 0 | 0 |
| F09 | aeotool | G3 free plans and trials | 2 | 2 | 2 |
| F10 | agentinterdict | G3 free plans and trials | 0 | 2 | 1 |
| F11 | blitz_cloud | G3 free plans and trials | 2 | 0 | 0 |
| F12 | anvevoice | G3 free plans and trials | 1 | 2 | 2 |
| F13 | 8nabler_voiceai | G4 usage pricing or missing prices | 2 | 0 | 0 |
| F14 | academic_intel_mcp | G4 usage pricing or missing prices | 2 | 0 | 2 |
| F15 | 12port | G4 usage pricing or missing prices | 2 | 0 | 0 |
| F16 | api_hub | G4 usage pricing or missing prices | 0 | 2 | 2 |
| F17 | 0xarchive_mcp | G5 mixed pricing and distractors | 2 | 0 | 0 |
| F18 | business_studio | G5 mixed pricing and distractors | 0 | 2 | 2 |
| F19 | aida_marketing | G5 mixed pricing and distractors | 0 | 1 | 0 |
| F20 | agent_utilities_2 | G5 mixed pricing and distractors | 2 | 2 | 0 |

## Notes

- Manifest: prompt sha256 1a381224174d4b4b1e8c89f03b3c4154dc3f7d22f99757d18fd420be29ccb7af; fixtures sha256 d476659a62df0bbf6c53179b8e96e19793c9e103699360a81ae88440d1c63d41; reference sha256 12e07198b4159b0f65d6abb6db2da8745daf27345d9a2bfa311015eedec4cbc1; 120 result rows.
- Counts, not rankings: 20 purposive products, paired by product across the two repetitions; no significance claims.
- Names and URLs are easy (they appear in the source metadata); the harder fields are free_tier, subscription and billing period — see the hard-fields chart and the known-value vs abstention columns.
