in reviewhumanPR #353Trust

Let verify load plugin files, and say when a validator is missing

A Lab scenario can name a validator that a plugin file registers (nandatown run scenario.yaml --plugin my_plugin.py). The run works and writes a bundle. nandatown verify on that bundle then fails with:

Author

Status
In review
Opened on
Nov 11
Branch
verify-plugin-flag

Description

The pitch.

## Trigger

A Lab scenario can name a validator that a plugin file registers (`nandatown run scenario.yaml --plugin my_plugin.py`). The run works and writes a bundle. `nandatown verify` on that bundle then fails with:

```
evaluator replay mismatch for evaluator ...
```

`verify` never loads plugin files, so the evaluator replay runs without the validator. The message reads as a tampered bundle when the bundle is intact.

## Change

- `nandatown verify --plugin FILE` (repeatable) loads plugin files before verifying, the same way `run --plugin` does. A file that cannot be loaded exits 2 with `problem: cannot load plugin file: ...`.
- When a Lab bundle names a validator that is not registered in the verifying process, the replay stage now says so and points at `--plugin`, instead of reporting a mismatch.
- One README sentence describing the flag.

`verify` still never loads a file because a bundle names it. The verifier chooses which plugin code to trust, on the command line.

## Compatibility

Bundles that use only built-in validators verify exactly as before: same stages, same messages, same exit codes. The evaluator version is unchanged. The only changed output is the detail text of the replay stage in the case that already failed.

## Verification

Three tests added to `tests/test_harness_overrides.py`, each building a real bundle from a scenario with a plugin validator:

- `test_verify_names_a_validator_it_cannot_replay`: without the plugin, the replay stage fails and names the validator and `--plugin`. Before the change this reported "evaluator replay mismatch".
- `test_verify_plugin_flag_replays_a_plugin_validator`: with `--plugin`, the bundle verifies.
- `test_verify_plugin_flag_reports_an_unloadable_file`: a missing plugin file exits 2.

Full suite: `python -m pytest -q`, Python 3.13: 1,864 passed, 2 skipped. Two tests in `tests/test_url_credentials.py` (`test_a_read_only_home_does_not_stop_a_run` and `test_pulse_history_is_readable_from_a_home_that_cannot_hold_a_key`) fail in my environment with and without this change, because the tests ran as root, which ignores read-only permissions. `git diff --check` is clean. No schema changes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Try it

Open PR on GitHubView diff

Checkout locally

git fetch origin pull/353/head:pr-353
git checkout pr-353