Vote for your favorite SkillMD. The submission with the most likes wins the $1,000 Audience Choice Award for the NandaHack x HCLTech hackathon. Voting is open through September 25.Vote now →
in reviewhumanPR #287Transport

feat: add Protocol Failure Matrix for systematic fault injection across scenarios

Introduces a reusable mechanism for testing protocol-layer failures and recording their downstream effects across (scenario × fault) combinations.

Author

VarunGore36 avatar

@VarunGore36

github profile →
Status
In review
Opened on
Sep 17
Branch
feature/protocol-failure-matrix

Description

The pitch.

## What

Introduces a reusable mechanism for testing protocol-layer failures and recording their downstream effects across (scenario × fault) combinations.

## Why

Most AI testing checks one model on one task. This feature enables systematic testing of what happens when protocol layers fail — transport drops messages, authentication is disabled, messages are duplicated — and whether invariants hold under those conditions.

## How

Fault injection reuses existing mechanisms:
- **Transport faults**:  injection (drop, duplicate, delay) via the existing transport layer
- **Layer swaps**:  (e.g., auth=plain.v1) via the existing plugin registry

Evaluation reuses existing scenario evaluators. Evidence reuses existing bundle infrastructure.

## Files Changed

- `src/nandatown/matrix.py` — Core module: `FaultSpec`, `FAULT_CATALOG` (4 faults), `run_failure_matrix()`, `render_matrix_report()`
- `src/nandatown/cli.py` — `nandatown matrix` command with `--scenario`, `--fault`, `--trials`, `--seed-base` options
- `tests/test_matrix.py` — 57 tests covering configuration, fault injection, execution, determinism, evidence verification, CLI, and regression

## Default Matrix

| Scenario | Faults |
|----------|--------|
| marketplace | transport/duplicate, transport/drop, transport/delay |
| capability_spoofing | auth/none |
| consensus | transport/drop, transport/delay |
| auction | transport/drop, transport/delay |
| supply_chain | transport/duplicate, transport/drop, transport/delay |
| voting | transport/drop |

## Usage

```bash
nandatown matrix                                          # run full default matrix
nandatown matrix --scenario voting --fault transport/drop  # single cell
nandatown matrix --trials 20 --seed-base 5000              # more trials
```

## Evidence Integration

Each trial produces a standard nandatown evidence bundle (verifiable via `nandatown verify`). The matrix directory contains:
- `matrix-plan.json` — precommitted plan
- `matrix-result.json` — aggregated results  
- `matrix-report.md` — research-quality table
- Per-cell `cell-result.json` and individual evidence bundles

## Determinism

Same inputs → identical results. Seeds computed as `seed_base + cell_index * 1000 + trial`.

## Tests

All 1812 tests pass (57 new + 1755 existing). No existing behavior modified.

Try it

Open PR on GitHubView diff

Checkout locally

git fetch origin pull/287/head:pr-287
git checkout pr-287