mergedhumanPR #264Identity
fix(evaluator): fail Track runs with duplicate or uncorrelated quote responses
One of nine independent PRs from a correctness audit of main at df0b5f1. Each merges cleanly with the others in any order; together they pass 908 tests locally on Python 3.11 and 3.14 (CI covers 3.11 and 3.12).
Description
The pitch.
One of nine independent PRs from a correctness audit of `main` at `df0b5f1`. Each merges cleanly with the others in any order; together they pass 908 tests locally on Python 3.11 and 3.14 (CI covers 3.11 and 3.12).
Commits: `47f72af`, `b351393`, `bd35174`, `3129bc1`, `f6403e4`, `f2ddf9c`
### Problem
Track evaluator 0.2.0 took the first accepted `quote_response` anywhere in the
run. It never counted responses or checked which request a response answered.
On `df0b5f1`, three Town-authored fixture sellers got a verified PASSED
(`nandatown verify` exit 0):
- one sent two responses;
- one sent a fresh response per duplicate delivery;
- one replied with `request_id: "q-does-not-exist"`.
### Change
- Evaluator 0.3.0 requires exactly one distinct quote response, whose
`request_id` names the accepted request.
- A second distinct response, a different id, or a non-string id (including
JSON null) fails `response` and `correct`.
- A missing `request_id` is not enough evidence (incomplete).
- Idempotent resends of the same message identity are not counted.
- The coordinator records the body's `request_id` on newly recorded
`message_accepted` events:
- a string of up to 128 characters is recorded verbatim;
- anything else is recorded as `{type, json_length, fingerprint}`.
- Correlation is still judged exactly, by comparing fingerprints. Stage notes
show at most 80 characters of any value.
- When a buyer sent several requests, the note names both counts instead of
implying the seller duplicated.
### Compatibility
- `nandatown verify` replays 0.2.0 bundles under the unchanged 0.2.0 rules.
Across 2,511 evaluations over 81 real and mutated bundles, output is
byte-identical to `df0b5f1`. Two genuine 0.2.0 bundles are committed as
fixtures. Unknown versions are still reported as a version difference.
- The skill tells sellers to carry "the request id", and every Town seller and
example sends it as `body.request_id`. The README now names the key. A
third-party seller that omits it gets incomplete instead of passed.
- The event-detail addition doesn't change the schemas (regenerated and
compared). `MANIFEST.in` ships the new fixtures.
- Not addressed, and a separate policy question: a 0.2.0 pass can still earn
TOWN-TESTED within its freshness window, because the badge doesn't name the
evaluator version.
### Verification
- Full suite on the branch: 853 passed.
- All eight stock profiles pass and verify. So do 40 of 40 campaign trials of
the fault profiles.
- Town-authored fixture sellers that send a second response (`dupresp`,
`uuidresp`) or name the wrong request (`norequestid`) now fail, run over
localhost through the real runner.
- A 200 KB `request_id`:
| File | Before | After |
|---|---|---|
| `result.json` | 401 KB | 1.5 KB |
| `events.jsonl` | 203 KB | 2.7 KB |
| `report.md` | 602 KB | 2.1 KB |
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Try it
Open PR on GitHubView diffCheckout locally
git fetch origin pull/264/head:pr-264
git checkout pr-264