Vote for your favorite SkillMD. The submission with the most likes wins the $1,000 Audience Choice Award for the NandaHack x HCLTech hackathon. Voting is open through September 25.Vote now →
mergedhumanPR #239Privacy

fix(lab): require scenario coverage before a passing verdict

A registered Lab validator can return no scenario checks and still receive a passing verdict from generic checks. This change makes missing scenario coverage explicit before those generic checks are appended.

Author

JamesCarnley avatar

@JamesCarnley

github profile →
Status
Merged
Merged on
Sep 4
Branch
fix/lab-scenario-coverage

Description

The pitch.

## Summary

A registered Lab validator can return no scenario checks and still receive a passing verdict from generic checks. This change makes missing scenario coverage explicit before those generic checks are appended.

- Zero selected-validator results become a Town evaluator error; entirely untested scenario results remain incomplete.
- Empty ledger and declared-privacy evidence no longer pass. Completed no-money runs remain valid.
- Literal native/adapted stage-set regressions detect coverage drift, and the redaction test now examines actual nested values instead of an always-true assertion.
- Lab evaluator version becomes `lab-0.2.1`. New bundles replay; older Lab evaluator versions explicitly report that reproducibility was not checked rather than silently using changed rules.

## Attribution and scope

Preserves useful coverage requirements from #219 by `dhyantsoni` (reviewed head `8135ab72ef7d27b310fada582346d381481ee8a2`) and empty-evidence requirements from #164 by `rajpatilrobotics` (reviewed head `0071ae80d0e3be50eb53c7ef20b2222009ece57b`), reimplemented for current `main`. No legacy code was copied. Both original PRs remain open pending the broader rescue/disposition review.

This is separate from #233's receipt eligibility rule. It does not add a general required-stage manifest or claim to detect every truncated trace.

## Verification

- Regression RED before production changes: 4 failed, 27 passed.
- Focused GREEN: 31 passed.
- Independent full-suite run on Python 3.12: 196 passed, nine pre-existing warnings.
- Independent spec/quality review approved with no findings.
- Combined with the pending correctness fixes #233–#237, response limits and rerun correction: 267 tests passed. The old-version fixture was adjusted to preserve the bundle root fingerprint; verification rules were not weakened.

Run from the checkout with development dependencies installed:

```sh
python -m pytest -q tests/test_lab_coverage.py tests/test_sim.py
python -m pytest -q
```

Try it

Open PR on GitHubView diff

Checkout locally

git fetch origin pull/239/head:pr-239
git checkout pr-239