mergedhumanPR #279Data Facts
fix(evaluator): require boolean acknowledgement flags and judge every accepted request
Two pre-existing evaluator gaps, reported against #264 and confirmed to
Author
@JamesCarnley
github profile →- Status
- Merged
- Merged on
- Sep 14
- Branch
- fix/track-evaluator-verdict-strictness
Description
The pitch.
Two pre-existing evaluator gaps, reported against #264 and confirmed to
reproduce on `main` at `df0b5f1`. They are not regressions from that PR or any
of #264–#278.
> **Stacked on #264.** This bumps the Track evaluator that #264 introduces, so
> it contains #264's commits and cannot merge before it. Only the last two
> commits are new here: `4e70266`, `b9ac153`, `fd50ec0` and `a9650f1`.
### The gaps
**A flag that says no was read as yes.** Acknowledgement flags were read for
truthiness, so a seller acknowledging `{"applied": "false"}` and a buyer
acknowledging `{"correct": "false"}` both counted as positive and the run
passed. The integer `1` passed too. Only an actual boolean `false` was read as
a no — the one spelling a participant is least likely to reach for by
accident was the only one that worked.
**A request nobody answered did not affect the verdict.** Only the first
accepted `quote_request` was ever judged. A buyer sending `q-1` and `q-2` with
only `q-1` answered came out `passed`, with `q-2` accepted, recorded, and
never looked at. The verdict spoke for the whole run while describing only the
part of it that went well.
Reproduced through the real coordinator with ordinary `TownClient` send,
claim and ack calls and untouched exported events. Five controls, on
`df0b5f1`:
| control | before | after |
|---|---|---|
| real booleans | passed | passed |
| `applied`/`correct` boolean `false` | failed | failed |
| `applied`/`correct` string `"false"` | **passed** | incomplete |
| `applied`/`correct` integer `1` | **passed** | incomplete |
| second request never answered | **passed** | incomplete |
### Change
- **A flag counts only when it is a JSON boolean.** `applied`, `correct` and
`duplicate` are all read that way. Anything else states nothing this
evaluator can read, and the stage note shows what was recorded so an
operator can see what to fix.
- **Inconclusive, not failed.** Town did not observe the work fail; it failed
to learn whether the work happened. Those are different findings, and
collapsing them would blame a subject for an outcome nobody observed.
- **A count is read the same way.** The fault stages compared
`tool_errors` and `context_truncations` with `>=`, which raises on a
string rather than answering. A participant's note is unvalidated, so a
seller acknowledging `{"applied": true, "tool_errors": "many"}` is accepted
by the coordinator and then makes the evaluator raise `TypeError` — Town
appearing to fail on evidence it holds, for a note the subject chose. A
count now has to be an integer, and a boolean is not one.
- **Every accepted request is judged.** One never claimed, acknowledged,
applied or answered leaves the stage it did not reach inconclusive, naming
that request, so the run is incomplete rather than passed. A stage that
already found something more specific about the first request keeps that
finding.
### Worth knowing
- A run with **more than one accepted response** is already a hard failure
under #264's rule that the profile expects exactly one. So a buyer that
asks twice and is answered twice fails, while one answered once is now
incomplete: the new rule only ever moves a verdict toward `incomplete`,
and there is no multi-request success path.
- A non-boolean `correct` alongside a response mismatch now reports
`not_enough_evidence` where `0.3.0` reported `failed`. That is deliberate —
an unreadable flag is not an observed wrong answer — and it is stricter for
the badge, because the stage then counts as not tested.
- A request addressed to someone other than the seller leaves the seller's
own stages alone, since it was never the seller's to handle, and is
reported under `response`, so the run is incomplete rather than passed.
- One readable `applied: true` beside an unreadable application claim does
not establish application exactly once. Two real `true`s still fail, and
the stock seller's duplicate acknowledgement, which says nothin
…Try it
Open PR on GitHubView diffCheckout locally
git fetch origin pull/279/head:pr-279
git checkout pr-279