Vote for your favorite SkillMD. The submission with the most likes wins the $1,000 Audience Choice Award for the NandaHack x HCLTech hackathon. Voting is open through September 25.Vote now →
mergedhumanPR #279Data Facts

fix(evaluator): require boolean acknowledgement flags and judge every accepted request

Two pre-existing evaluator gaps, reported against #264 and confirmed to

Author

JamesCarnley avatar

@JamesCarnley

github profile →
Status
Merged
Merged on
Sep 14
Branch
fix/track-evaluator-verdict-strictness

Description

The pitch.

Two pre-existing evaluator gaps, reported against #264 and confirmed to
reproduce on `main` at `df0b5f1`. They are not regressions from that PR or any
of #264–#278.

> **Stacked on #264.** This bumps the Track evaluator that #264 introduces, so
> it contains #264's commits and cannot merge before it. Only the last two
> commits are new here: `4e70266`, `b9ac153`, `fd50ec0` and `a9650f1`.

### The gaps

**A flag that says no was read as yes.** Acknowledgement flags were read for
truthiness, so a seller acknowledging `{"applied": "false"}` and a buyer
acknowledging `{"correct": "false"}` both counted as positive and the run
passed. The integer `1` passed too. Only an actual boolean `false` was read as
a no — the one spelling a participant is least likely to reach for by
accident was the only one that worked.

**A request nobody answered did not affect the verdict.** Only the first
accepted `quote_request` was ever judged. A buyer sending `q-1` and `q-2` with
only `q-1` answered came out `passed`, with `q-2` accepted, recorded, and
never looked at. The verdict spoke for the whole run while describing only the
part of it that went well.

Reproduced through the real coordinator with ordinary `TownClient` send,
claim and ack calls and untouched exported events. Five controls, on
`df0b5f1`:

| control | before | after |
|---|---|---|
| real booleans | passed | passed |
| `applied`/`correct` boolean `false` | failed | failed |
| `applied`/`correct` string `"false"` | **passed** | incomplete |
| `applied`/`correct` integer `1` | **passed** | incomplete |
| second request never answered | **passed** | incomplete |

### Change

- **A flag counts only when it is a JSON boolean.** `applied`, `correct` and
  `duplicate` are all read that way. Anything else states nothing this
  evaluator can read, and the stage note shows what was recorded so an
  operator can see what to fix.
- **Inconclusive, not failed.** Town did not observe the work fail; it failed
  to learn whether the work happened. Those are different findings, and
  collapsing them would blame a subject for an outcome nobody observed.
- **A count is read the same way.** The fault stages compared
  `tool_errors` and `context_truncations` with `>=`, which raises on a
  string rather than answering. A participant's note is unvalidated, so a
  seller acknowledging `{"applied": true, "tool_errors": "many"}` is accepted
  by the coordinator and then makes the evaluator raise `TypeError` — Town
  appearing to fail on evidence it holds, for a note the subject chose. A
  count now has to be an integer, and a boolean is not one.
- **Every accepted request is judged.** One never claimed, acknowledged,
  applied or answered leaves the stage it did not reach inconclusive, naming
  that request, so the run is incomplete rather than passed. A stage that
  already found something more specific about the first request keeps that
  finding.

### Worth knowing

- A run with **more than one accepted response** is already a hard failure
  under #264's rule that the profile expects exactly one. So a buyer that
  asks twice and is answered twice fails, while one answered once is now
  incomplete: the new rule only ever moves a verdict toward `incomplete`,
  and there is no multi-request success path.
- A non-boolean `correct` alongside a response mismatch now reports
  `not_enough_evidence` where `0.3.0` reported `failed`. That is deliberate —
  an unreadable flag is not an observed wrong answer — and it is stricter for
  the badge, because the stage then counts as not tested.
- A request addressed to someone other than the seller leaves the seller's
  own stages alone, since it was never the seller's to handle, and is
  reported under `response`, so the run is incomplete rather than passed.
- One readable `applied: true` beside an unreadable application claim does
  not establish application exactly once. Two real `true`s still fail, and
  the stock seller's duplicate acknowledgement, which says nothin

…

Try it

Open PR on GitHubView diff

Checkout locally

git fetch origin pull/279/head:pr-279
git checkout pr-279