in reviewhumanPR #164Registry
rajpatilrobotics: fail closed on empty scenario traces
NANDA Town is a protocol test rig, so a successful validation must be backed by actual trace evidence.
Author
@rajpatilrobotics
github profile →- Status
- In review
- Opened on
- Jul 10
- Branch
- hackathon/rajpatilrobotics-empty-trace-validation
Description
The pitch.
## Summary
NANDA Town is a protocol test rig, so a successful validation must be backed by actual trace evidence.
This change adds a shared fail-closed invariant: every registered scenario now rejects an empty event trace with:
- `name`: `trace_nonempty`
- `passed`: `false`
- `detail`: `trace contains no events and therefore cannot prove the scenario invariants`
## Why this matters
Before this change, 9 of 22 registered scenarios returned only passing results for an empty event list:
`comms_versioning`, `marketplace`, `auction`, `voting`, `consensus`,
`supply_chain`, `reputation`, `streaming_payments`, and `empic_payments`.
A zero-byte marketplace JSONL file therefore produced three passing validator
results, and `validate_protocol()` reported `all_passed=true`.
From my distributed edge, IoT, and LoRa experience, missing telemetry is not
evidence of successful operation. A silent or incorrectly generated trace must
not certify protocol invariants.
## Root cause
`validate_trace()` loads a zero-byte file as an empty list and delegates to
`validate_events()`. The shared dispatcher previously invoked scenario
validators immediately. Several validators looped over zero events and passed
through vacuous conditions.
## Design
The invariant is enforced once in `validate_events()`, after distinguishing
unknown scenario names and before invoking scenario-specific validators.
This keeps the change:
- deterministic and dependency-free;
- shared across `validate_events()`, `validate_trace()`, and
`validate_protocol()`;
- compatible with existing unknown-scenario behavior;
- compatible with all valid non-empty result names, details, and ordering;
- independent of individual scenario validator semantics.
## Testing
Added coverage for:
- every registered scenario through a parameterized registry test;
- an empty event list through `validate_events()`;
- a zero-byte JSONL file through `validate_trace()`;
- `validate_protocol()` serialization and `all_passed=false`;
- empty and non-empty unknown scenarios;
- an existing valid marketplace trace retaining its original passing results.
Test-first evidence before implementation:
24 failed, 142 deselected
After implementation:
24 passed, 142 deselected
166 targeted tests passed
Full repository gate:
make ci-local
1174 passed, 1 skipped, 1 deselected
Ruff clean
Ruff formatting clean
Pyright: 0 errors
## Compatibility and limitations
Missing files and malformed JSON already fail by exception. This change handles
the global empty-evidence case. Syntactically valid but non-empty incomplete
traces still depend on scenario-specific evidence validators.
Unknown scenario names preserve their existing behavior.
Try it
Open PR on GitHubView diffCheckout locally
git fetch origin pull/164/head:pr-164
git checkout pr-164