Vote for your favorite SkillMD. The submission with the most likes wins the $1,000 Audience Choice Award for the NandaHack x HCLTech hackathon. Voting is open through September 25.Vote now →
mergedhumanPR #281Other

fix(evaluator): 0.5.0 binds duplicate delivery to the injected offer and lets only a terminal buyer verdict decide

Follow-up to the review of merged 00a013b. Two of its findings change what the

Author

JamesCarnley avatar

@JamesCarnley

github profile →
Status
Merged
Merged on
Sep 16
Branch
fix/track-evaluator-0.5.0

Description

The pitch.

Follow-up to the review of merged `00a013b`. Two of its findings change what the
Track evaluator concludes, and both are fixed here under one new version,
`0.5.0`, as the maintainer chose. A third problem, with how replay chooses a
version's rules, had to be fixed first.

Each was reproduced on `main` through the real coordinator before any change.

### 1. Moving the current version quietly rewrote older ones

`0.4.0` decided which of its rules applied by asking whether the version being
replayed was the current one. That held only while `0.4.0` stayed current: the
release that moved `EVALUATOR_VERSION` forward would have replayed every
recorded `0.4.0` bundle under `0.3.0`'s rules. An `applied: "false"` run
recorded as incomplete would have replayed as passed.

The rules each version applies are now a table keyed by version, and a replay
looks up the version the bundle recorded. Releasing a version adds a row and
changes no earlier one. This commit changes no behaviour, and a test releases a
pretend next version to prove no older judgment moves.

### 2. Duplicate delivery was not bound to the delivery it injects

`quote-duplicate-delivery` checks that a seller recognises the duplicate the
town injects by re-offering completed work. A seller that lost its lease first
was redelivered the request, and since #278 it acknowledges that redelivery as
`{duplicate: true, applied: true}`. Nothing told that acknowledgement apart
from one of the injected delivery:

- the runner took it as the seller finished and stopped **before the town had
  offered the duplicate at all**, so a correct seller came out incomplete;
- the evaluator paired it with a later offer the seller claimed and never
  acknowledged, and **passed** the run.

The coordinator now records the injected offer's `fence` and `attempt` on
`duplicate_offered`. The runner counts the duplicate as handled, and `0.5.0`
recognises it, only from a `processed` acknowledgement under that fence. An
offer recorded without a fence cannot be bound, and `0.5.0` does not guess.

### 3. A provisional buyer assertion decided the verdict

A buyer acknowledging `retryable` with `correct: true` and then settling with
`correct: false` **passed**, because the first assertion was taken; the reverse
order failed. The maintainer decided the rule: only a terminal acknowledgement,
any status but `retryable`, decides `correct`. A provisional assertion neither
outweighs one nor stands in for one that is missing, so a buyer that only
asserted provisionally, or settled without asserting, leaves `correct`
inconclusive. Terminal assertions that disagree about one response are
inconclusive whichever order they arrived in.

This moves verdicts both ways, deliberately: a provisional doubt followed by a
terminal "correct" now passes, because the buyer's settled conclusion is what
the stage asks for.

### Compatibility

- **Replay of `0.2.0`, `0.3.0` and `0.4.0` is exact.** Checked against genuine
  sources extracted from git (`df0b5f1`, `4810d3a`, `main`) across 81,437 event
  logs and real bundles recorded by each version, including a `0.4.0`
  duplicate-delivery bundle whose offer has no fence.
- **Every legitimate seller tried still passes:** stock, LLM, A2A bridge,
  `examples/byoa_seller.py`, external, after one or two lost leases, and after a
  crash-restart. Runs with a lost lease or a crash-restart now **pass** where
  `main` stopped early and reported them incomplete.
- **Under `0.5.0`, verdicts move only toward `incomplete`,** apart from the
  provisional-then-terminal cases above. That covers an offer acknowledged with
  any status but `processed`, and disagreeing terminal assertions.
- **`duplicate_offered` gains two fields.** The coordinator HTTP contract and
  the profiles are otherwise unchanged.
- **Shipped-versions list, with #283.** Both PRs carry an identical
  `SHIPPED_EVALUATOR_VERSIONS` block naming Track `0.5.0`, so they merge
  cleanly in either order and neither needs amending after the 

…

Try it

Open PR on GitHubView diff

Checkout locally

git fetch origin pull/281/head:pr-281
git checkout pr-281