OOBench

Headline finding

4,800 tokens of dated documents move accuracy from 0.520 to 0.533.

That is the evidence lift, measured against a forced closed-book control that is not allowed to abstain: +0.013, with the two 95% intervals — [0.393, 0.667] and [0.38, 0.66] — lying on top of each other. Two explanations survive and this benchmark does not yet separate them: the resolver may not be using the documents, or the documents may not contain the answer. The separate calibration failure is sharper — 36% of the answers the model rates ≥0.9 confident are wrong, over 122 confident answers.

Problem statement

Almost every LLM evaluation grades itself: a human judges the output, or a second model does. The benchmark then inherits whatever bias the judge has, and confident wrong answers — the interesting failures — are exactly what a model-judge is worst at catching.

Approach

Score resolver output against settled Polymarket / UMA markets, so ground truth is how the market actually paid out — externally settled and financially incentivised rather than a judgement call. 50 questions were drawn from 4,947 harvested candidates through a published funnel, every rejection recorded with its reason, and each question ships with 2–5 dated Wikipedia revisions pinned by revid so the same bundle returns the same bytes in a year.

The domain's real base rate was measured rather than assumed — the screened pool settles 72% NO — then forced to 20 YES / 20 NO, so the naive always_no baseline scores 0.500 and is printed on the same line as every model result.

Every resolver call must return a fixed schema — outcome, confidence, reasoning, sources, conflicts detected — validated before it is scored. A malformed response gets one repair retry carrying the validation error, then a recorded hard_fail: a measured failure rather than a crash. Whether a cited source id actually exists in the bundle is checked outside the schema, because a fabricated id is well-formed and hallucinated citations deserve their own metric.

Results — gemini-3.5-flash-lite, 50 questions × 3 arms × 3 repeats

The result is negative, and it is published as it stands. No arm separates from a coin flip — A0's 95% interval on accuracy is [0.393, 0.667], which contains 0.500. Neither prompt change helped: McNemar against A0 gives p=1.0 for A1 and p=0.63 for A2 on 3–4 discordant questions, and the smallest difference this dataset can detect is about 11 points. The differences in the table are 1–3 points, so they are not results. The closed_book row is in the same table, and its 0.020 is almost entirely abstention: without documents the model declines on 96% of questions rather than guessing.

Reliability diagram — A0

Predicted probability against observed frequency, five equal-count bins. The diagonal is perfect calibration; everything below it is overconfidence. The top bin is the whole finding — the model predicts 1.00 and is right 0.63 of the time.

Base → change → delta

One change, reported with its cost: giving the resolver an explicit UNRESOLVABLE option (A0 → A2).

Calibration improves and instability follows it: A2 buys its ECE with a self-consistency drop. Overall accuracy still falls, because an abstention is scored as wrong — both numbers are true and the table shows the one the plan specified. McNemar gives p=0.63 on 4 discordant questions, so this is reported as not significant, with the 11-point minimum detectable effect stated alongside it.

The more interesting failure is where the model abstains. On the contested set — markets where UMA token holders actually disputed the outcome — A0 abstains on 10% of questions against 28.3% on the main set. A resolver that understood its own uncertainty would do the opposite.

Engineering metrics — absent from the comparable work, and the point here

• Citation grounding 1.000 across every scored resolution — the model never once cited a source id that was not in its bundle.
• $0.002 per resolution, p50 1.5 s / p95 2.6 s, priced from each call's reported usage at list rates. Cache hit rate is 0.000: this model does not use the implicit cache at any prompt length sent.
• Structured outputs are deliberately not used. The API can constrain the response to the schema, which would make schema-valid, repair and hard-fail read 1.0 / 0 / 0 by construction — three reliability numbers deleted by the feature that appears to guarantee them. The contract is stated in the prompt and the model has to meet it.
• CI gate replays 182 recorded cassettes, so ruff, mypy --strict and pytest run on every pull request with no API key and no network. A captured 429 is re-raised on replay, which exercises the backoff path for free. A nightly live run re-records and opens an issue on provider drift rather than failing the build — drift is a finding, not a broken commit.