mergedhumanPR #241Registry
fix(lab): disclose untested scope of imported scenarios
Imported legacy scenarios run Town reference agents, not their original plugins or validators. Previously a declared network_partition disappeared silently, and a generic adapted exchange could pass without any result saying that the…
Description
The pitch.
## Summary Imported legacy scenarios run Town reference agents, not their original plugins or validators. Previously a declared `network_partition` disappeared silently, and a generic adapted exchange could pass without any result saying that the original protocol was untested. - Disclose every unsupported `failures` key in the exported profile and report, including zero/disabled declarations. Unsupported payload values are not copied into these notes. - Add `original_scenario: not_tested` to adapted results, with no supporting event IDs. Keep the existing generic local-flow checks and their outcomes. - Label the human verdict as applying to the adapted reference flow only. - Bump the Lab evaluator to `lab-0.2.2`; old bundles report a version mismatch instead of silently acquiring different checks. No new dependency, profile schema field, execution backend, payment system or partition simulator is introduced. Existing `message_drop` behavior, including its disclosed 0.2 cap, remains unchanged. Original agent configuration, plugin code, and validators are still not executed. ## Source ideas and scope This current-main change was motivated by the actual-run discipline in @stanleyoz's #153 and the declared-condition/non-vacuity checks in @ang101's #190. Their tests highlighted why completing a substituted marketplace flow must not be presented as proving streaming payments or gossip isolation. This is a focused coverage/disclosure replacement, not a port of either original implementation. Streaming lifecycle idempotency, cancellation/refunds and conservation require a separately selected streaming scenario. Per-node partition isolation and legitimate bridge relay require an actual partition-capable registry fixture. Those remain useful future suites, not capabilities established by this PR. No source code from either legacy PR was copied. ## Verification - Clean baseline: 236 passed. - Regression-first checks reproduced missing unsupported-condition notes and missing original-scenario coverage before the fix. - Full local suite: 248 passed, 9 pre-existing warnings, Python 3.12. - Actual `run_lab` round trips for synthetic streaming and gossip imports: reference task passes, original scenario remains untested, omissions survive export, and bundles verify/replay. - Empty adapted trace remains incomplete; native scenario stage expectations and existing fault controls remain green. - A genuine pre-change `lab-0.2.1` bundle was generated and verified before the edit; the new code retains its recorded verdict and reports the expected evaluator-version mismatch. - Self-reviewed against the bounded requirements; no new independent-agent review was performed for this batch. Reproduce with `python -m pytest -q`. CI also runs Python 3.11/3.12 and the existing Lab/Track smoke commands.
Try it
Open PR on GitHubView diffCheckout locally
git fetch origin pull/241/head:pr-241
git checkout pr-241