Vote for your favorite SkillMD. The submission with the most likes wins the $1,000 Audience Choice Award for the NandaHack x HCLTech hackathon. Voting is open through September 25.Vote now →
in reviewhumanPR #164Registry

rajpatilrobotics: fail closed on empty scenario traces

NANDA Town is a protocol test rig, so a successful validation must be backed by actual trace evidence.

Author

rajpatilrobotics avatar

@rajpatilrobotics

github profile →
Status
In review
Opened on
Jul 10
Branch
hackathon/rajpatilrobotics-empty-trace-validation

Description

The pitch.

## Summary

NANDA Town is a protocol test rig, so a successful validation must be backed by actual trace evidence.

This change adds a shared fail-closed invariant: every registered scenario now rejects an empty event trace with:

- `name`: `trace_nonempty`
- `passed`: `false`
- `detail`: `trace contains no events and therefore cannot prove the scenario invariants`

## Why this matters

Before this change, 9 of 22 registered scenarios returned only passing results for an empty event list:

`comms_versioning`, `marketplace`, `auction`, `voting`, `consensus`,
`supply_chain`, `reputation`, `streaming_payments`, and `empic_payments`.

A zero-byte marketplace JSONL file therefore produced three passing validator
results, and `validate_protocol()` reported `all_passed=true`.

From my distributed edge, IoT, and LoRa experience, missing telemetry is not
evidence of successful operation. A silent or incorrectly generated trace must
not certify protocol invariants.

## Root cause

`validate_trace()` loads a zero-byte file as an empty list and delegates to
`validate_events()`. The shared dispatcher previously invoked scenario
validators immediately. Several validators looped over zero events and passed
through vacuous conditions.

## Design

The invariant is enforced once in `validate_events()`, after distinguishing
unknown scenario names and before invoking scenario-specific validators.

This keeps the change:

- deterministic and dependency-free;
- shared across `validate_events()`, `validate_trace()`, and
  `validate_protocol()`;
- compatible with existing unknown-scenario behavior;
- compatible with all valid non-empty result names, details, and ordering;
- independent of individual scenario validator semantics.

## Testing

Added coverage for:

- every registered scenario through a parameterized registry test;
- an empty event list through `validate_events()`;
- a zero-byte JSONL file through `validate_trace()`;
- `validate_protocol()` serialization and `all_passed=false`;
- empty and non-empty unknown scenarios;
- an existing valid marketplace trace retaining its original passing results.

Test-first evidence before implementation:

    24 failed, 142 deselected

After implementation:

    24 passed, 142 deselected
    166 targeted tests passed

Full repository gate:

    make ci-local
    1174 passed, 1 skipped, 1 deselected
    Ruff clean
    Ruff formatting clean
    Pyright: 0 errors

## Compatibility and limitations

Missing files and malformed JSON already fail by exception. This change handles
the global empty-evidence case. Syntactically valid but non-empty incomplete
traces still depend on scenario-specific evidence validators.

Unknown scenario names preserve their existing behavior.

Try it

Open PR on GitHubView diff

Checkout locally

git fetch origin pull/164/head:pr-164
git checkout pr-164