Vote for your favorite SkillMD. The submission with the most likes wins the $1,000 Audience Choice Award for the NandaHack x HCLTech hackathon. Voting is open through October 25.Vote now →
in reviewhumanPR #310Other

fix(evaluator): do not pass `correct` on an assertion the run contradicts

…icts

Author

rahulver551 avatar

@rahulver551

github profile →
Status
In review
Opened on
Oct 9
Branch
fix/correct-contradicted-by-record

Description

The pitch.

…icts

A seller that quoted 3999 cents for the 3990-cent task, and a buyer that asserted `correct: true` without comparing, produced a run in which every stage passed, the verdict was `passed`, and `nandatown verify` reported that the evaluator reproduces the recorded result. The bundle contradicts itself: the buyer's terminal acknowledgement records `{"correct": true, "total_cents": 3999, "expected_total_cents": 3990}` and the seller's processed acknowledgement records `total_cents: 3999`. Under 0.5.0 `correct` is decided by the buyer's boolean alone. Reproduced through the real runner with a fixture seller and a fixture buyer.

Under 0.6.0 ("consistent_verdict") a `correct: true` passes only when nothing the run recorded contradicts it:

- the buyer's own terminal acknowledgement recording `total_cents`, or `expected_total_cents`, other than the task's expected total fails `correct`: that record is the buyer's, and it sits beside the assertion;
- a seller's processed acknowledgement recording another `total_cents` leaves `correct` inconclusive: the town does not record the response's total, so the two participants' records disagree and neither decides.

Totals are read only as JSON integers, the way counts already are; a string, float, boolean or container states nothing and changes nothing. A `correct: false` is judged as before. 0.2.0 to 0.5.0 bundles replay on the assertion alone, unchanged; an existing 0.5.0 bundle of the reproduction still verifies.

Tests: tests/test_correct_contradicted.py (16), observed failing on main first (4 failed), including one through `run_town` with a new `wrongtotal` mode on tests/fixtures/track_seller_variant.py and a new tests/fixtures/credulous_buyer.py. Full suite: 1768 passed, 2 skipped; the two read-only-home tests in test_url_credentials.py fail only when run as root and pass as an ordinary user, as on main.

Not included: SHIPPED_EVALUATOR_VERSIONS in bundle.py, whose comment says to add a version when it merges.

Try it

Open PR on GitHubView diff

Checkout locally

git fetch origin pull/310/head:pr-310
git checkout pr-310