Vote for your favorite SkillMD. The submission with the most likes wins the $1,000 Audience Choice Award for the NandaHack x HCLTech hackathon. Voting is open through September 25.Vote now →
mergedhumanPR #283Data Facts

docs: say which historical bundles replay, and which cannot

Lower-priority follow-up to the review of merged 00a013b: statements about

Author

JamesCarnley avatar

@JamesCarnley

github profile →
Status
Merged
Merged on
Sep 16
Branch
docs/historical-replay-accuracy

Description

The pitch.

Lower-priority follow-up to the review of merged `00a013b`: statements about
replaying historical evidence that had drifted from what `verify` and `receipt`
do. Every claim below was checked against bundles older Towns really wrote, and
each is now pinned by a test.

### What was wrong

- **Receipts overclaimed.** The docs said a bundle "recorded by a known earlier
  evaluator version" is accepted without replay and with a disclosure. That
  holds only where this Town can no longer replay the version, such as an older
  Lab evaluator or `path-0.1`. Every Track version, the current Lab evaluator
  `lab-0.2.6`, and each Path profile from `path-0.2` *are* replayed, and the
  receipt rests on that replay. `make_receipt`, `verify_receipt`,
  `verify_bundle_integrity`, the disclosure constant and the mirror section all
  repeated the old claim.
- **Track `0.2.0` replay has exceptions nobody wrote down.** Two early commits
  changed what `0.2.0` records without a new version:
  - `6e7529b` (24 August 2026) reworded the `portable_identity` note. **Every**
    bundle from an earlier build is affected.
  - `f46edce` (25 August 2026) began recording a stage that lacks evidence after
    a failed stage as `not_tested` ("not reached") instead of
    `not_enough_evidence`. Affected: **failing runs** from earlier builds in
    which such a stage followed the failure.

  In both cases the verdict replays unchanged. But `verify` compares the whole
  result, so it reports an evaluator replay mismatch, and `receipt` refuses the
  bundle. The architecture notes now state the rule this implies: any change to
  what an evaluator records needs a new version.
- **The shipped-versions list was stale.** It claimed to name every version
  `main` recorded, but listed only Track `0.2.0`; `0.3.0` and `0.4.0` have since
  merged. It is consulted only for a bundle this Town cannot replay, so adding
  them changes no current result. Its comment now says that, instead of
  claiming receipts accept listed versions without replay.
- **A docstring still described the old receipt.** `bundle_receipt_check` said
  the disclosure was command output only, but #266 has signed it into the
  receipt since.

### Change

Documentation, code comments and the shipped-versions list only. There is no
behavioural change. A replay rule for the early `0.2.0` builds would change
evaluator meaning, so it is left for a maintainer.

**Shipped-versions list, with #281.** Both PRs carry an identical
`SHIPPED_EVALUATOR_VERSIONS` block naming Track `0.5.0`, so they merge cleanly
in either order and neither needs amending after the other lands. They are
meant to merge together: this PR alone would list `0.5.0` before #281 ships it,
and accept a bundle claiming it for a receipt without replay instead of
refusing it as unrecognised.

### Verification

- **Two genuine early `0.2.0` fixtures,** kept verbatim with provenance:
  - a passing run recorded by the `7a6a00c` build, differing from replay in the
    `portable_identity` note only;
  - a failing run recorded by `22b8984`, the parent of `f46edce`, with a
    deliberately faulty local seller. It differs in one unreached stage's status
    and note, and its verdict still replays as `failed`.
- **The boundary:** the independent reviewer replayed 6,010 evaluator cases
  across all five early fault profiles under the evaluators of `d32d3e1`,
  `6adfd18`, `7a6a00c`, `6e7529b`, `c86bed2`, `22b8984`, `f46edce` and
  `df0b5f1`:
  - builds before `6e7529b` differ in the note and the cascade;
  - builds from `6e7529b` to `22b8984` differ in the cascade only;
  - `f46edce` and `df0b5f1` match exactly;
  - no verdict differs anywhere.
- **Other historical bundles:** genuine `path-0.1`, `path-0.2`, `path-0.3` and
  `lab-0.2.0`–`lab-0.2.6` bundles, verified with this code.
- **Full suite:** 1,576 passed.
- **Independent review, three rounds.** Corrections that came from review:
  - the affected range was too narrow;
  - the claim that statuses replay u

…

Try it

Open PR on GitHubView diff

Checkout locally

git fetch origin pull/283/head:pr-283
git checkout pr-283