in reviewhumanPR #287Transport
feat: add Protocol Failure Matrix for systematic fault injection across scenarios
Introduces a reusable mechanism for testing protocol-layer failures and recording their downstream effects across (scenario × fault) combinations.
Author
@VarunGore36
github profile →- Status
- In review
- Opened on
- Sep 17
- Branch
- feature/protocol-failure-matrix
Description
The pitch.
## What Introduces a reusable mechanism for testing protocol-layer failures and recording their downstream effects across (scenario × fault) combinations. ## Why Most AI testing checks one model on one task. This feature enables systematic testing of what happens when protocol layers fail — transport drops messages, authentication is disabled, messages are duplicated — and whether invariants hold under those conditions. ## How Fault injection reuses existing mechanisms: - **Transport faults**: injection (drop, duplicate, delay) via the existing transport layer - **Layer swaps**: (e.g., auth=plain.v1) via the existing plugin registry Evaluation reuses existing scenario evaluators. Evidence reuses existing bundle infrastructure. ## Files Changed - `src/nandatown/matrix.py` — Core module: `FaultSpec`, `FAULT_CATALOG` (4 faults), `run_failure_matrix()`, `render_matrix_report()` - `src/nandatown/cli.py` — `nandatown matrix` command with `--scenario`, `--fault`, `--trials`, `--seed-base` options - `tests/test_matrix.py` — 57 tests covering configuration, fault injection, execution, determinism, evidence verification, CLI, and regression ## Default Matrix | Scenario | Faults | |----------|--------| | marketplace | transport/duplicate, transport/drop, transport/delay | | capability_spoofing | auth/none | | consensus | transport/drop, transport/delay | | auction | transport/drop, transport/delay | | supply_chain | transport/duplicate, transport/drop, transport/delay | | voting | transport/drop | ## Usage ```bash nandatown matrix # run full default matrix nandatown matrix --scenario voting --fault transport/drop # single cell nandatown matrix --trials 20 --seed-base 5000 # more trials ``` ## Evidence Integration Each trial produces a standard nandatown evidence bundle (verifiable via `nandatown verify`). The matrix directory contains: - `matrix-plan.json` — precommitted plan - `matrix-result.json` — aggregated results - `matrix-report.md` — research-quality table - Per-cell `cell-result.json` and individual evidence bundles ## Determinism Same inputs → identical results. Seeds computed as `seed_base + cell_index * 1000 + trial`. ## Tests All 1812 tests pass (57 new + 1755 existing). No existing behavior modified.
Try it
Open PR on GitHubView diffCheckout locally
git fetch origin pull/287/head:pr-287
git checkout pr-287