mergedhumanPR #237Other
fix(runner): reject unknown agent override roles
Reject unsupported role keys in both harnesses and external before starting a Track run.
Description
The pitch.
## Summary Reject unsupported role keys in both `harnesses` and `external` before starting a Track run. Previously, `--agent seler=cmd:/does/not/exist` silently left the stock seller selected, produced a passing run, and retained the ignored override in the evidence. A typo could therefore look like a successful test of the user's own agent even though that command never ran. The runner validates against the selected profile's roles immediately after profile lookup, before allocating ports, creating directories or credentials, invoking callbacks, or starting processes. The CLI reports the unknown and supported roles with exit 2. It catches only the new narrow `RunnerUsageError`, preserving the distinction from runtime failures. Valid overrides, harness-over-external precedence, external handoff, and stock defaults are unchanged. ## Regression coverage - Typo and empty roles in each mapping are rejected. - Mixed valid/invalid mappings and `external` values of `None` are rejected when their key is invalid. - Startup sentinels and absent output directories confirm rejection precedes side effects. - CLI returns a usage error rather than a successful reference-agent run. - Existing real external-command and LLM harness tests remain passing. ## Verification - `.venv/bin/python -m pytest -q tests/test_harness_overrides.py`: 14 passed. - `.venv/bin/python -m pytest -q`: 190 passed. - `.venv/bin/nandatown run marketplace --out <temporary-dir>`: PASSED. - `.venv/bin/nandatown run quote-crash-restart --out <temporary-dir>`: PASSED. - `git diff --check`: passed. - Python 3.12.13 locally; current CI also exercises Python 3.11. CI has no separate lint/format step. - TDD first reproduced the missing rejection against the real runner. Independent expert correctness and vision review completed. - Combined integration with #233 and the other three correctness fixes: 213 tests passed on both Python 3.11.15 and 3.12.13. Base: `projnanda/nandatown:main` at `4d57012a3dfb6b7aeee5ab429b26513a5eef4505`. No new harness or role model is introduced.
Try it
Open PR on GitHubView diffCheckout locally
git fetch origin pull/237/head:pr-237
git checkout pr-237