closedhumanPR #219Registry
fix(metrics): don't report zero validators as a passing verdict; guard registry drift
validate_events does VALIDATORS.get(scenario_type, []), and metrics.validate_protocol reported all_passed = all(r.passed for r in vresults).
Author
@dhyantsoni
github profile →- Status
- Closed without merging
- Opened on
- Aug 14
- Branch
- fix/vacuous-validation-pass-and-registry-conformance
Description
The pitch.
## The bug
`validate_events` does `VALIDATORS.get(scenario_type, [])`, and `metrics.validate_protocol` reported `all_passed = all(r.passed for r in vresults)`.
`all([])` is `True`. So an unregistered or misspelled scenario type produced a green verdict having executed **zero** checks:
```python
>>> from nest_core.metrics import validate_protocol
>>> validate_protocol("traces/marketplace.jsonl", "definitely_not_a_real_scenario_type")
{'validations': [], 'all_passed': True}
```
`validate_protocol` is the public, documented API a leaderboard or CI would call. As written, a typo in `task.type` is indistinguishable from a clean pass.
Seven shipped scenarios currently reach this path with no registry entry — `gossip_registry`, `gossip_eclipse`, `gossip_byzantine_forgery`, `gossip_signed_equivocation`, `delegated_auth`, `delegated_auth_partition`, `capability_spoofing`.
**To be precise about scope:** several of those *are* covered by bespoke validators under `nest_plugins_reference/validators/` that their own tests call directly. This is **not** a claim that they are unverified. It is a claim that through `validate_protocol` they reported success having run nothing.
## The fix
`all_passed` is now true only when the scenario type is registered **and** at least one validator ran, with a `validators_run` count and an explicit `unknown_scenario_type` flag so callers can distinguish "checked nothing" from "checked everything and passed":
```python
>>> validate_protocol("traces/marketplace.jsonl", "typo_here")
{'validations': [], 'validators_run': 0, 'unknown_scenario_type': True, 'all_passed': False}
```
`validate_protocol` has no in-tree callers besides its own docstring, so the behaviour change is contained.
## Why the existing guard didn't catch it
`test_validators.py::TestValidatorRegistry::test_all_scenario_types_registered` asserts only that a *name* is a key in `VALIDATORS`. It never asks whether a scenario of that type can be built or run — so a name can satisfy that test while being unreachable end to end.
`test_registry_conformance.py` cross-references the three registries that are maintained by hand with nothing enforcing agreement:
1. `scenarios/*.yaml` — shipped scenarios, each naming a `task.type`
2. `nest_core.scenarios` — the `_try_load_builtin` if/elif dispatch chain
3. `nest_core.validators` — the `VALIDATORS` mapping
## What it found at `675a20b`
Marked `xfail(strict=True)` with the specific reason, so **CI stays green** while the gap stays visible. Strict means fixing one turns it into an `XPASS` failure — the signal to delete the marker rather than let it rot.
**Two shipped scenarios cannot run at all.** `scenarios/streaming_payments.yaml` and `scenarios/streaming_payments_partition.yaml` declare `task.type: streaming_payments`, for which no factory exists anywhere in the tree:
```console
$ uv run nest run scenarios/streaming_payments.yaml --seed 42
KeyError: "No scenario factory registered for 'streaming_payments'"
```
Added by #21. The `streaming` payments **plugin** is real and registered — it's the scenarios demonstrating it that are dead.
**Six validators are registered but unreachable:**
| Key | Validators | Why |
|---|---|---|
| `streaming_payments` | `validate_streaming_conservation`, `validate_streaming_no_drain_after_close`, `validate_streaming_no_overbill_on_partition` | scenarios have no factory (above) |
| `receipt_reputation_majority` | `validate_receipt_reputation_honest_confidence`, `validate_receipt_reputation_ring_majority`, `validate_receipt_reputation_ring_severed` | no scenario declares this type — `receipt_reputation_majority_ring.yaml` declares `task.type: receipt_reputation` |
They're unit-tested against hand-built event fixtures (`test_validators.py:773-870`, `2084-2092`) and have never executed against a trace from a running scenario.
I've deliberately **not** silenced these. Resolving them needs maintainer decisions — whether to write the missing `streaming_payments` f
…Try it
Open PR on GitHubView diffCheckout locally
git fetch origin pull/219/head:pr-219
git checkout pr-219