in reviewhumanPR #310Other
fix(evaluator): do not pass `correct` on an assertion the run contradicts
…icts
Author
@rahulver551
github profile →- Status
- In review
- Opened on
- Oct 9
- Branch
- fix/correct-contradicted-by-record
Description
The pitch.
…icts
A seller that quoted 3999 cents for the 3990-cent task, and a buyer that asserted `correct: true` without comparing, produced a run in which every stage passed, the verdict was `passed`, and `nandatown verify` reported that the evaluator reproduces the recorded result. The bundle contradicts itself: the buyer's terminal acknowledgement records `{"correct": true, "total_cents": 3999, "expected_total_cents": 3990}` and the seller's processed acknowledgement records `total_cents: 3999`. Under 0.5.0 `correct` is decided by the buyer's boolean alone. Reproduced through the real runner with a fixture seller and a fixture buyer.
Under 0.6.0 ("consistent_verdict") a `correct: true` passes only when nothing the run recorded contradicts it:
- the buyer's own terminal acknowledgement recording `total_cents`, or `expected_total_cents`, other than the task's expected total fails `correct`: that record is the buyer's, and it sits beside the assertion;
- a seller's processed acknowledgement recording another `total_cents` leaves `correct` inconclusive: the town does not record the response's total, so the two participants' records disagree and neither decides.
Totals are read only as JSON integers, the way counts already are; a string, float, boolean or container states nothing and changes nothing. A `correct: false` is judged as before. 0.2.0 to 0.5.0 bundles replay on the assertion alone, unchanged; an existing 0.5.0 bundle of the reproduction still verifies.
Tests: tests/test_correct_contradicted.py (16), observed failing on main first (4 failed), including one through `run_town` with a new `wrongtotal` mode on tests/fixtures/track_seller_variant.py and a new tests/fixtures/credulous_buyer.py. Full suite: 1768 passed, 2 skipped; the two read-only-home tests in test_url_credentials.py fail only when run as root and pass as an ordinary user, as on main.
Not included: SHIPPED_EVALUATOR_VERSIONS in bundle.py, whose comment says to add a version when it merges.Try it
Open PR on GitHubView diffCheckout locally
git fetch origin pull/310/head:pr-310
git checkout pr-310