Vote for your favorite SkillMD. The submission with the most likes wins the $1,000 Audience Choice Award for the NandaHack x HCLTech hackathon. Voting is open through September 25.Vote now →
mergedhumanPR #264Identity

fix(evaluator): fail Track runs with duplicate or uncorrelated quote responses

One of nine independent PRs from a correctness audit of main at df0b5f1. Each merges cleanly with the others in any order; together they pass 908 tests locally on Python 3.11 and 3.14 (CI covers 3.11 and 3.12).

Author

JamesCarnley avatar

@JamesCarnley

github profile →
Status
Merged
Merged on
Sep 14
Branch
fix/track-response-correlation

Description

The pitch.

One of nine independent PRs from a correctness audit of `main` at `df0b5f1`. Each merges cleanly with the others in any order; together they pass 908 tests locally on Python 3.11 and 3.14 (CI covers 3.11 and 3.12).

Commits: `47f72af`, `b351393`, `bd35174`, `3129bc1`, `f6403e4`, `f2ddf9c`

### Problem

Track evaluator 0.2.0 took the first accepted `quote_response` anywhere in the
run. It never counted responses or checked which request a response answered.
On `df0b5f1`, three Town-authored fixture sellers got a verified PASSED
(`nandatown verify` exit 0):

- one sent two responses;
- one sent a fresh response per duplicate delivery;
- one replied with `request_id: "q-does-not-exist"`.

### Change

- Evaluator 0.3.0 requires exactly one distinct quote response, whose
  `request_id` names the accepted request.
- A second distinct response, a different id, or a non-string id (including
  JSON null) fails `response` and `correct`.
- A missing `request_id` is not enough evidence (incomplete).
- Idempotent resends of the same message identity are not counted.
- The coordinator records the body's `request_id` on newly recorded
  `message_accepted` events:
  - a string of up to 128 characters is recorded verbatim;
  - anything else is recorded as `{type, json_length, fingerprint}`.
- Correlation is still judged exactly, by comparing fingerprints. Stage notes
  show at most 80 characters of any value.
- When a buyer sent several requests, the note names both counts instead of
  implying the seller duplicated.

### Compatibility

- `nandatown verify` replays 0.2.0 bundles under the unchanged 0.2.0 rules.
  Across 2,511 evaluations over 81 real and mutated bundles, output is
  byte-identical to `df0b5f1`. Two genuine 0.2.0 bundles are committed as
  fixtures. Unknown versions are still reported as a version difference.
- The skill tells sellers to carry "the request id", and every Town seller and
  example sends it as `body.request_id`. The README now names the key. A
  third-party seller that omits it gets incomplete instead of passed.
- The event-detail addition doesn't change the schemas (regenerated and
  compared). `MANIFEST.in` ships the new fixtures.
- Not addressed, and a separate policy question: a 0.2.0 pass can still earn
  TOWN-TESTED within its freshness window, because the badge doesn't name the
  evaluator version.

### Verification

- Full suite on the branch: 853 passed.
- All eight stock profiles pass and verify. So do 40 of 40 campaign trials of
  the fault profiles.
- Town-authored fixture sellers that send a second response (`dupresp`,
  `uuidresp`) or name the wrong request (`norequestid`) now fail, run over
  localhost through the real runner.
- A 200 KB `request_id`:

| File | Before | After |
|---|---|---|
| `result.json` | 401 KB | 1.5 KB |
| `events.jsonl` | 203 KB | 2.7 KB |
| `report.md` | 602 KB | 2.1 KB |

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Try it

Open PR on GitHubView diff

Checkout locally

git fetch origin pull/264/head:pr-264
git checkout pr-264