in reviewhumanPR #354Identity

KnownTruth: add qualification-gate buyer, MCP-edge candidate seller and refusal test

KnownTruth builds a test environment from an organization's records with the correct answers planted by construction, runs candidate agents on it, grades them against the planted truth, and serves the results to other agents over MCP.

Author

a-weckstein avatar

@a-weckstein

github profile →
Status
In review
Opened on
Nov 11
Branch
integration/knowntruth-qualification-gate

Description

The pitch.

### Summary

KnownTruth builds a test environment from an organization's records with the correct answers planted by construction, runs candidate agents on it, grades them against the planted truth, and serves the results to other agents over MCP.

This PR adds a Town example, `examples/known_truth/`, in which:

- a buyer admits a seller only when the project's qualification card shows that the seller's agent passed every exact check on the known-truth benchmark, and refuses otherwise;
- the admitted agent is a real Claude tool-loop agent (the project's own candidate) playing seller through Town's MCP participant adapter, `nandatown mcp serve`;
- Town's evaluator and evidence bundle record what happened under a duplicate-delivery fault.

### Integration

| Town interface | How the project uses it |
|---|---|
| Track mailbox (join, participants, messages, claim with fence, ack) | Two independently written standard-library participants: `qualifying_buyer.py` and `candidate_seller.py` |
| MCP participant adapter (`nandatown mcp serve`) | Tool surface for the project's Claude candidate agent in the seller role (`candidate_seller.py --mode live`) |
| MCP probe (`nandatown mcp test`) | Handshake and tool listing against the project's evaluation MCP server |

The project itself is a separate, private repository. What this PR shares from it is the qualification cards (`examples/known_truth/cards/`), exported from a saved evaluation of three candidate agents. A card holds public fields only (candidate, counts, names of failed checks, pass or fail) and a SHA-256 over its body. The buyer recomputes the hash and fails closed on a mismatch.

Scripted mode (`--mode scripted`, the default) uses no model. It proves the gate mechanics and lets Town's CI run the test without credentials. It is not agent evidence. Live mode needs the project checkout and the operator's own model key.

### Reproduce

- Town commit: `6f9a7c0` (evaluator 0.5.0, nandatown 0.2.0)
- Python: 3.11
- Project version: private repository; identified here by the saved evaluation run `live_03` (the source of the cards) and the SHA-256 prefixes of its two Town-facing files: `town_candidate.py` `e6087de4c3013c15`, `export_cards.py` `edf11bdfe2f4ba0a`
- Live agent: the project's `haiku55_medium` candidate (Claude Haiku 5.5, medium effort), through `nandatown mcp serve`

```bash
python3.11 -m venv .venv && .venv/bin/python -m pip install -e '.[dev]'
```

Offline test (no credentials, loopback only, about 19 s):

```bash
.venv/bin/python -m pytest -q tests/test_known_truth_example.py
```

Each case by hand (`CARD` is a file in `examples/known_truth/cards/`):

```bash
nandatown run PROFILE --agent buyer=cmd:"python examples/known_truth/qualifying_buyer.py --card examples/known_truth/cards/CARD" --agent seller=cmd:"python examples/known_truth/candidate_seller.py --mode scripted"
```

Recorded live evidence, checked offline:

```bash
nandatown verify examples/known_truth/evidence/run-9310c30f4b28
```

The live command (needs the project checkout and the operator's own model key) is in `examples/known_truth/README.md`.

### Results and limitations

| Case | Profile | Expected | Observed |
|---|---|---|---|
| Qualifying card, scripted seller | `quote-duplicate-delivery` | PASSED; `duplicate_recognized` and `correct` passed; the buyer's acknowledgement note carries the card reference; `verify` clean | As expected |
| Not-qualifying card, scripted seller | `quote-clean` | No quote request sent; refusal `not_qualified` naming the failed checks; buyer exit 6 | As expected; run INCOMPLETE at `accepted`, about 9 s |
| Tampered card, scripted seller | `quote-clean` | Refusal `card_hash_mismatch`; buyer exit 7 | As expected; run INCOMPLETE, about 9 s |
| Qualifying card, live Claude seller, baseline | `quote-duplicate-delivery` | PASSED | **INCOMPLETE.** The agent answered 3990 once and acknowledged, then ended its turn. `duplicate_recognized` was inconclusive: Town never got to offer the red

…

Try it

Open PR on GitHubView diff

Checkout locally

git fetch origin pull/354/head:pr-354
git checkout pr-354