Vote for your favorite SkillMD. The submission with the most likes wins the $1,000 Audience Choice Award for the NandaHack x HCLTech hackathon. Voting is open through September 25.Vote now →
mergedhumanPR #223Registry

feat(agent-test): evaluate external agents in Town

A successful adapter exchange is not enough: Town must drive the participant through a real deterministic workflow and emit honest, stage-specific evidence instead of a single opaque score.

Author

JamesCarnley avatar

@JamesCarnley

github profile →
Status
Merged
Merged on
Sep 2
Branch
feat/town-agent-test-engine

Description

The pitch.

## Problem

A successful adapter exchange is not enough: Town must drive the participant through a real deterministic workflow and emit honest, stage-specific evidence instead of a single opaque score.

## Change

- run one external participant through the pinned Town scenario and run-local Registry
- evaluate five ordered stages with pass/fail/incomplete/not-evaluated distinctions
- emit private, no-overwrite `result.json` and `trace.jsonl` artifacts
- close admitted runs and artifacts across success, failure, interruption, and Town defects

## Verification

- final stack: `make ci-local` — all 5 checks passed; 2,081 passed, 2 skipped, 1 deselected
- runner, evaluator, lifecycle, artifact-permission, collision, cancellation, and exact-byte tests pass
- independent lifecycle/security review found no Critical or Important issues

## Stack

**4 of 9. Depends on PR 3.** PR 5 adds the first user journey; this PR is the evaluation engine.

**Merge note:** After #222 merges into `main`, retarget this PR to `main`, wait for its CI, and only then merge it.

Try it

Open PR on GitHubView diff

Checkout locally

git fetch origin pull/223/head:pr-223
git checkout pr-223