Measured, not marketed
Pilots and notes with the numbers attached. Every post says what was measured, how, and what it does not show.
Which open-weight models can extract prices accurately? A wider JohnRouter test
155 open-weight chat models screened on 20 frozen pricing pages, five finalists validated twice on 20 fresh pages, against a closed control. The best open-weight models matched the control; most of the catalogue did not, and a large share of failures were formatting rather than facts. 3,360 scored calls, US$9.45, no JSON mode, no repairs.
What three AI models got wrong on 20 pricing pages
Two open-weight models through JohnRouter against the closed model that reads pricing pages in production today. 120 calls, deterministic scoring, every charge logged, reproduction package. Completely correct records: 26, 21 and 17 of 40 (65.0%, 52.5%, 42.5%). Preliminary; three reference answers disputed.