mergedhumanPR #223Registry
feat(agent-test): evaluate external agents in Town
A successful adapter exchange is not enough: Town must drive the participant through a real deterministic workflow and emit honest, stage-specific evidence instead of a single opaque score.
Description
The pitch.
## Problem A successful adapter exchange is not enough: Town must drive the participant through a real deterministic workflow and emit honest, stage-specific evidence instead of a single opaque score. ## Change - run one external participant through the pinned Town scenario and run-local Registry - evaluate five ordered stages with pass/fail/incomplete/not-evaluated distinctions - emit private, no-overwrite `result.json` and `trace.jsonl` artifacts - close admitted runs and artifacts across success, failure, interruption, and Town defects ## Verification - final stack: `make ci-local` — all 5 checks passed; 2,081 passed, 2 skipped, 1 deselected - runner, evaluator, lifecycle, artifact-permission, collision, cancellation, and exact-byte tests pass - independent lifecycle/security review found no Critical or Important issues ## Stack **4 of 9. Depends on PR 3.** PR 5 adds the first user journey; this PR is the evaluation engine. **Merge note:** After #222 merges into `main`, retarget this PR to `main`, wait for its CI, and only then merge it.
Try it
Open PR on GitHubView diffCheckout locally
git fetch origin pull/223/head:pr-223
git checkout pr-223