closedhumanPR #219Registry

fix(metrics): don't report zero validators as a passing verdict; guard registry drift

validate_events does VALIDATORS.get(scenario_type, []), and metrics.validate_protocol reported all_passed = all(r.passed for r in vresults).

Author

Status
Closed without merging
Opened on
Aug 14
Branch
fix/vacuous-validation-pass-and-registry-conformance

Description

The pitch.

## The bug

`validate_events` does `VALIDATORS.get(scenario_type, [])`, and `metrics.validate_protocol` reported `all_passed = all(r.passed for r in vresults)`.

`all([])` is `True`. So an unregistered or misspelled scenario type produced a green verdict having executed **zero** checks:

```python
>>> from nest_core.metrics import validate_protocol
>>> validate_protocol("traces/marketplace.jsonl", "definitely_not_a_real_scenario_type")
{'validations': [], 'all_passed': True}
```

`validate_protocol` is the public, documented API a leaderboard or CI would call. As written, a typo in `task.type` is indistinguishable from a clean pass.

Seven shipped scenarios currently reach this path with no registry entry — `gossip_registry`, `gossip_eclipse`, `gossip_byzantine_forgery`, `gossip_signed_equivocation`, `delegated_auth`, `delegated_auth_partition`, `capability_spoofing`.

**To be precise about scope:** several of those *are* covered by bespoke validators under `nest_plugins_reference/validators/` that their own tests call directly. This is **not** a claim that they are unverified. It is a claim that through `validate_protocol` they reported success having run nothing.

## The fix

`all_passed` is now true only when the scenario type is registered **and** at least one validator ran, with a `validators_run` count and an explicit `unknown_scenario_type` flag so callers can distinguish "checked nothing" from "checked everything and passed":

```python
>>> validate_protocol("traces/marketplace.jsonl", "typo_here")
{'validations': [], 'validators_run': 0, 'unknown_scenario_type': True, 'all_passed': False}
```

`validate_protocol` has no in-tree callers besides its own docstring, so the behaviour change is contained.

## Why the existing guard didn't catch it

`test_validators.py::TestValidatorRegistry::test_all_scenario_types_registered` asserts only that a *name* is a key in `VALIDATORS`. It never asks whether a scenario of that type can be built or run — so a name can satisfy that test while being unreachable end to end.

`test_registry_conformance.py` cross-references the three registries that are maintained by hand with nothing enforcing agreement:

1. `scenarios/*.yaml` — shipped scenarios, each naming a `task.type`
2. `nest_core.scenarios` — the `_try_load_builtin` if/elif dispatch chain
3. `nest_core.validators` — the `VALIDATORS` mapping

## What it found at `675a20b`

Marked `xfail(strict=True)` with the specific reason, so **CI stays green** while the gap stays visible. Strict means fixing one turns it into an `XPASS` failure — the signal to delete the marker rather than let it rot.

**Two shipped scenarios cannot run at all.** `scenarios/streaming_payments.yaml` and `scenarios/streaming_payments_partition.yaml` declare `task.type: streaming_payments`, for which no factory exists anywhere in the tree:

```console
$ uv run nest run scenarios/streaming_payments.yaml --seed 42
KeyError: "No scenario factory registered for 'streaming_payments'"
```

Added by #21. The `streaming` payments **plugin** is real and registered — it's the scenarios demonstrating it that are dead.

**Six validators are registered but unreachable:**

| Key | Validators | Why |
|---|---|---|
| `streaming_payments` | `validate_streaming_conservation`, `validate_streaming_no_drain_after_close`, `validate_streaming_no_overbill_on_partition` | scenarios have no factory (above) |
| `receipt_reputation_majority` | `validate_receipt_reputation_honest_confidence`, `validate_receipt_reputation_ring_majority`, `validate_receipt_reputation_ring_severed` | no scenario declares this type — `receipt_reputation_majority_ring.yaml` declares `task.type: receipt_reputation` |

They're unit-tested against hand-built event fixtures (`test_validators.py:773-870`, `2084-2092`) and have never executed against a trace from a running scenario.

I've deliberately **not** silenced these. Resolving them needs maintainer decisions — whether to write the missing `streaming_payments` f

…

Try it

Open PR on GitHubView diff

Checkout locally

git fetch origin pull/219/head:pr-219
git checkout pr-219