Vote for your favorite SkillMD. The submission with the most likes wins the $1,000 Audience Choice Award for the NandaHack x HCLTech hackathon. Voting is open through September 25.Vote now →
mergedhumanPR #241Registry

fix(lab): disclose untested scope of imported scenarios

Imported legacy scenarios run Town reference agents, not their original plugins or validators. Previously a declared network_partition disappeared silently, and a generic adapted exchange could pass without any result saying that the…

Author

JamesCarnley avatar

@JamesCarnley

github profile →
Status
Merged
Merged on
Sep 4
Branch
fix/upstream-adaptation-scope

Description

The pitch.

## Summary

Imported legacy scenarios run Town reference agents, not their original plugins or validators. Previously a declared `network_partition` disappeared silently, and a generic adapted exchange could pass without any result saying that the original protocol was untested.

- Disclose every unsupported `failures` key in the exported profile and report, including zero/disabled declarations. Unsupported payload values are not copied into these notes.
- Add `original_scenario: not_tested` to adapted results, with no supporting event IDs. Keep the existing generic local-flow checks and their outcomes.
- Label the human verdict as applying to the adapted reference flow only.
- Bump the Lab evaluator to `lab-0.2.2`; old bundles report a version mismatch instead of silently acquiring different checks.

No new dependency, profile schema field, execution backend, payment system or partition simulator is introduced. Existing `message_drop` behavior, including its disclosed 0.2 cap, remains unchanged. Original agent configuration, plugin code, and validators are still not executed.

## Source ideas and scope

This current-main change was motivated by the actual-run discipline in @stanleyoz's #153 and the declared-condition/non-vacuity checks in @ang101's #190. Their tests highlighted why completing a substituted marketplace flow must not be presented as proving streaming payments or gossip isolation.

This is a focused coverage/disclosure replacement, not a port of either original implementation. Streaming lifecycle idempotency, cancellation/refunds and conservation require a separately selected streaming scenario. Per-node partition isolation and legitimate bridge relay require an actual partition-capable registry fixture. Those remain useful future suites, not capabilities established by this PR. No source code from either legacy PR was copied.

## Verification

- Clean baseline: 236 passed.
- Regression-first checks reproduced missing unsupported-condition notes and missing original-scenario coverage before the fix.
- Full local suite: 248 passed, 9 pre-existing warnings, Python 3.12.
- Actual `run_lab` round trips for synthetic streaming and gossip imports: reference task passes, original scenario remains untested, omissions survive export, and bundles verify/replay.
- Empty adapted trace remains incomplete; native scenario stage expectations and existing fault controls remain green.
- A genuine pre-change `lab-0.2.1` bundle was generated and verified before the edit; the new code retains its recorded verdict and reports the expected evaluator-version mismatch.
- Self-reviewed against the bounded requirements; no new independent-agent review was performed for this batch.

Reproduce with `python -m pytest -q`. CI also runs Python 3.11/3.12 and the existing Lab/Track smoke commands.

Try it

Open PR on GitHubView diff

Checkout locally

git fetch origin pull/241/head:pr-241
git checkout pr-241