in reviewhumanPR #353Trust
Let verify load plugin files, and say when a validator is missing
A Lab scenario can name a validator that a plugin file registers (nandatown run scenario.yaml --plugin my_plugin.py). The run works and writes a bundle. nandatown verify on that bundle then fails with:
Description
The pitch.
## Trigger A Lab scenario can name a validator that a plugin file registers (`nandatown run scenario.yaml --plugin my_plugin.py`). The run works and writes a bundle. `nandatown verify` on that bundle then fails with: ``` evaluator replay mismatch for evaluator ... ``` `verify` never loads plugin files, so the evaluator replay runs without the validator. The message reads as a tampered bundle when the bundle is intact. ## Change - `nandatown verify --plugin FILE` (repeatable) loads plugin files before verifying, the same way `run --plugin` does. A file that cannot be loaded exits 2 with `problem: cannot load plugin file: ...`. - When a Lab bundle names a validator that is not registered in the verifying process, the replay stage now says so and points at `--plugin`, instead of reporting a mismatch. - One README sentence describing the flag. `verify` still never loads a file because a bundle names it. The verifier chooses which plugin code to trust, on the command line. ## Compatibility Bundles that use only built-in validators verify exactly as before: same stages, same messages, same exit codes. The evaluator version is unchanged. The only changed output is the detail text of the replay stage in the case that already failed. ## Verification Three tests added to `tests/test_harness_overrides.py`, each building a real bundle from a scenario with a plugin validator: - `test_verify_names_a_validator_it_cannot_replay`: without the plugin, the replay stage fails and names the validator and `--plugin`. Before the change this reported "evaluator replay mismatch". - `test_verify_plugin_flag_replays_a_plugin_validator`: with `--plugin`, the bundle verifies. - `test_verify_plugin_flag_reports_an_unloadable_file`: a missing plugin file exits 2. Full suite: `python -m pytest -q`, Python 3.13: 1,864 passed, 2 skipped. Two tests in `tests/test_url_credentials.py` (`test_a_read_only_home_does_not_stop_a_run` and `test_pulse_history_is_readable_from_a_home_that_cannot_hold_a_key`) fail in my environment with and without this change, because the tests ran as root, which ignores read-only permissions. `git diff --check` is clean. No schema changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Try it
Open PR on GitHubView diffCheckout locally
git fetch origin pull/353/head:pr-353
git checkout pr-353