in reviewhumanPR #339Other
fix(verify): replay bundles judged by a plugin validator
A Lab scenario can bring its own validator through plugin_files (or nandatown run --plugin FILE), but nandatown verify replays the evaluator without loading it.
Description
The pitch.
A Lab scenario can bring its own validator through `plugin_files` (or `nandatown run --plugin FILE`), but `nandatown verify` replays the evaluator without loading it. The validator is then not registered, the replay differs from `result.json`, and every such bundle fails with "evaluator replay mismatch" although nothing in it changed. I ran into this while writing a knowledge-routing scenario with its own validator; it reproduces on `main` with the scenario from `nandatown new scenario demo` once `plugin_files` and `validator` are filled in. `verify` now takes `--plugin FILE` (repeatable), as `run` does, and loads the named files before the replay. It stays opt-in because loading a plugin runs its code: `verify` still never loads paths recorded inside a bundle. Without the flag, a mismatch on a Lab bundle whose validator is not registered now names that validator and the flag, and a plugin path that does not exist is refused with exit code 2. Otherwise the checks and exit codes are unchanged; Track and Path bundles are unaffected. The README gets one sentence next to `plugin_files`. `tests/test_verify_plugin.py` builds a scenario from the scaffold with a plugin validator, runs it, unregisters the validator to stand in for a fresh process, and checks `verify` without the plugin, with it, and with a missing file. The three tests fail on `main` and pass with this change. The full suite passes on Python 3.11 and 3.12 except two tests in `tests/test_url_credentials.py`, which fail the same way on `main` when the suite runs as root; `nandatown run marketplace` and `nandatown run quote-crash-restart` still pass and their bundles still verify. Written with help from Claude (see the co-author line in the commit).
Try it
Open PR on GitHubView diffCheckout locally
git fetch origin pull/339/head:pr-339
git checkout pr-339