Vote for your favorite SkillMD. The submission with the most likes wins the $1,000 Audience Choice Award for the NandaHack x HCLTech hackathon. Voting is open through September 25.Vote now →
mergedhumanPR #237Other

fix(runner): reject unknown agent override roles

Reject unsupported role keys in both harnesses and external before starting a Track run.

Author

JamesCarnley avatar

@JamesCarnley

github profile →
Status
Merged
Merged on
Sep 4
Branch
fix/reject-unknown-agent-roles

Description

The pitch.

## Summary

Reject unsupported role keys in both `harnesses` and `external` before starting a Track run.

Previously, `--agent seler=cmd:/does/not/exist` silently left the stock seller selected, produced a passing run, and retained the ignored override in the evidence. A typo could therefore look like a successful test of the user's own agent even though that command never ran.

The runner validates against the selected profile's roles immediately after profile lookup, before allocating ports, creating directories or credentials, invoking callbacks, or starting processes. The CLI reports the unknown and supported roles with exit 2. It catches only the new narrow `RunnerUsageError`, preserving the distinction from runtime failures.

Valid overrides, harness-over-external precedence, external handoff, and stock defaults are unchanged.

## Regression coverage

- Typo and empty roles in each mapping are rejected.
- Mixed valid/invalid mappings and `external` values of `None` are rejected when their key is invalid.
- Startup sentinels and absent output directories confirm rejection precedes side effects.
- CLI returns a usage error rather than a successful reference-agent run.
- Existing real external-command and LLM harness tests remain passing.

## Verification

- `.venv/bin/python -m pytest -q tests/test_harness_overrides.py`: 14 passed.
- `.venv/bin/python -m pytest -q`: 190 passed.
- `.venv/bin/nandatown run marketplace --out <temporary-dir>`: PASSED.
- `.venv/bin/nandatown run quote-crash-restart --out <temporary-dir>`: PASSED.
- `git diff --check`: passed.
- Python 3.12.13 locally; current CI also exercises Python 3.11. CI has no separate lint/format step.
- TDD first reproduced the missing rejection against the real runner. Independent expert correctness and vision review completed.
- Combined integration with #233 and the other three correctness fixes: 213 tests passed on both Python 3.11.15 and 3.12.13.

Base: `projnanda/nandatown:main` at `4d57012a3dfb6b7aeee5ab429b26513a5eef4505`. No new harness or role model is introduced.

Try it

Open PR on GitHubView diff

Checkout locally

git fetch origin pull/237/head:pr-237
git checkout pr-237