in reviewhumanPR #366Other

Graveyard: record what a fail-closed outside dependency costs under a lease

The Agentic Graveyard is a pre-mortem service for agents: a record of failures, each stored with the check that would have prevented it, served over MCP. This PR does not add the service.

Author

ostinatorigore42 avatar

@ostinatorigore42

github profile →
Status
In review
Opened on
Nov 12
Branch
integration/graveyard-failclosed

Description

The pitch.

## Summary

The Agentic Graveyard is a pre-mortem service for agents: a record of failures, each stored with the check that would have prevented it, served over MCP. This PR does not add the service. It adds a Town participant that *depends* on an outside service, and records what that dependency costs under a lease.

The useful result is not the integration. It is this: **a participant's timeout for an outside dependency must be strictly under its lease, or its refusal cannot be told apart from its death.** Nothing in `docs/` says so, and it is easy to get wrong.

## Integration

`examples/graveyard_seller.py` joins the HTTP mailbox as a seller (`test-agent --cmd`) and calls an external MCP endpoint before it applies a quote request. The lookup result goes into the acknowledgement note, so a bundle records whether the service was reached, whether it abstained, and what it cost.

- Standard library only, like `examples/byoa_seller.py`. No model credentials, no third-party package.
- The service is a separately hosted deployment (`agentic-graveyard.vercel.app`, MCP `2025-06-18`). Its source is not in this PR and is not needed to reproduce anything here.
- The check runs once per `message_id`, not once per delivery, so a duplicated delivery replays the stored response without paying for a second lookup inside the same lease.
- `GRAVEYARD_POLICY` selects fail-closed (default) or fail-open when the service is down.

## Reproduce

Town commit `6f9a7c064486`, `nandatown` 0.2.0, evaluator 0.5.0, Python 3.12.10, Windows 11. Service `agentic-graveyard` 1.1.0.

```bash
python3 -m venv .venv && source .venv/bin/activate
python -m pip install -e .
SELLER="python examples/graveyard_seller.py"

nandatown test-agent --role seller --cmd "$SELLER" --profile quote-clean
nandatown test-agent --role seller --cmd "$SELLER" --profile quote-duplicate-delivery
GRAVEYARD_URL=http://127.0.0.1:9/mcp nandatown test-agent --role seller --cmd "$SELLER" --profile quote-clean
GRAVEYARD_URL=http://127.0.0.1:9/mcp GRAVEYARD_POLICY=fail-open nandatown test-agent --role seller --cmd "$SELLER" --profile quote-clean
GRAVEYARD_URL=http://127.0.0.1:9/mcp nandatown test-agent --role seller --cmd "$SELLER" --profile quote-crash-restart
GRAVEYARD_URL=http://127.0.0.1:9/mcp GRAVEYARD_TIMEOUT=0.8 nandatown test-agent --role seller --cmd "$SELLER" --profile quote-crash-restart

python -m pytest -q tests/test_graveyard_seller.py
```

Port 9 refuses connections, so every failure case runs offline. Three bundles are included under `docs/integrations/evidence/` with `state/` removed; each passes `nandatown verify` from a fresh clone.

## Results and limitations

| # | Profile, lease | Service | Policy | Expected | Observed |
| --- | --- | --- | --- | --- | --- |
| 1 | `quote-clean`, 5 s | up | closed | passes, lookup recorded | **PASSED** 6/7, lookup 140 ms |
| 2 | `quote-duplicate-delivery`, 5 s | up | closed | applied once | **PASSED** 7/8, one application, response replayed |
| 3 | `quote-clean`, 5 s | down | closed | refuses | **INCOMPLETE** 2/7, 20 claims, 20 `retryable`, no response sent |
| 4 | `quote-clean`, 5 s | down | open | answers, marked degraded | **PASSED** 6/7, note carries `degraded: true` |
| 5 | `quote-crash-restart`, 1.5 s | down | closed | refusal recorded | **no acknowledgement at all**: 20 claims, 17 `claim_expired`, 20 `stale_fence_rejected`, **0 `ack_recorded`** |
| 6 | same, `GRAVEYARD_TIMEOUT=0.8` | down | closed | refusal recorded | 38 claims, **38 `ack_recorded`**, all `retryable`, 797-829 ms |

A reachable lookup costs 140-188 ms. A refused connection costs about 2.0 s on this platform whatever the configured timeout, because the connect attempt is what takes the time. The crash profiles run a 1.5 s lease. So in case 5 the lease expired while the participant was still waiting to learn its dependency was unreachable; its `retryable` acknowledgement then arrived against a dead fence and Town rejected it, correctly. The participant was refusing on 

…

Try it

Open PR on GitHubView diff

Checkout locally

git fetch origin pull/366/head:pr-366
git checkout pr-366