mergedhumanPR #15Transport
[Platform] Research harness for A/B hackathon experiments
Ships scripts/harness/ — multi-condition, reproducible experiment infrastructure to turn the hackathon into a publishable AI benchmark.
Description
The pitch.
## Summary
Ships `scripts/harness/` — multi-condition, reproducible experiment infrastructure to turn the hackathon into a publishable AI benchmark. Lets you run N>=100 agents per (model x brief_specificity x pre_push_checklist) cell and produce a dataset + plots the same way every time, from a plain shell or from inside Claude Code.
Everything is exercised by a default-suite pytest gate using mocked fixtures, so this PR adds zero live-agent cost while wiring up the path to spend that cost later.
## What lands
| file | purpose |
| --- | --- |
| `scripts/harness/conditions.yaml` | factor schema: `model` x `brief_specificity` x `pre_push_checklist`, plus a `skip:` list and per-cell defaults. Cartesian product produces 17 live cells (18 - 1 skipped). Each cell has a stable `cell_id` (12-char sha256 of canonical `(conditions_version, factors)` JSON), so the same factor combo always maps to the same id, on any machine, in any process. |
| `scripts/harness/conditions.py` | YAML loader + cell expansion + `compute_cell_id()`. |
| `scripts/harness/agent_runner.py` | Two transports: `FixtureAgentRunner` (dry-run, replays mocked submissions deterministically) and `ClaudeCLIAgentRunner` (live; shells out to `claude -p ... --output-format stream-json`). The CLI path was chosen over a hand-rolled Anthropic SDK tool-loop because it's simpler and reuses the CLI's existing file-edit / branch-hygiene plumbing — documented in `README.md`. |
| `scripts/harness/run_condition.py` | CLI: `--cell <cell_id> --n <N>`. Spawns N agents in isolated workdirs (`worktree` / `clone` / `ephemeral` strategies — `clone` is the most reproducible for use outside Claude Code), writes one JSONL line per submission to `data/hackathon-runs/<cell_id>.jsonl`, flushed and `fsync`'d after each row so a crash loses at most a partial line. Best-effort `gh`-CLI enrichment fills `head_sha`, `lines_added/removed`, `first_push_ci_status`, `iterations_to_green`. |
| `scripts/harness/collect.py` | Aggregates per-cell JSONLs into `data/hackathon-runs/all.jsonl`, deterministically sorted by `(cell_id, run_idx)`. Refuses to merge rows whose `schema_version` doesn't match the current harness — prevents silent schema drift. |
| `scripts/harness/analyze.py` | Three PNG plots: diversity collapse (top-1 / top-3 layer-cluster share per condition), calibration (claimed-CI-green vs actual on first push), iteration efficiency (pushes-to-green distribution). matplotlib is gated behind an optional `harness` extra in `pyproject.toml`, so the core repo stays matplotlib-free. |
| `scripts/harness/briefs/{vague,layer-enumerated,open-problems}.md` | The three brief templates wired to the `brief_specificity` factor. `open-problems.md` renders the `docs/hackathon/problems/` listing if it exists and falls back to the vague brief if the parallel open-problems track hasn't landed yet. |
| `scripts/harness/dry_run/fixtures/*.json` | 5 mocked agent submissions covering green / claimed-green-but-actually-red / iterating-to-green / spawn-failure cases. |
| `scripts/harness/dry_run/test_dry_run.py` | Default-suite pytest gate: runs `run_condition` over fixtures for 2 cells x 4 replicates, asserts the JSONL row schema is exactly the 29 documented fields, runs `collect.py` and asserts sorted ordering + schema-mismatch refusal, runs `analyze.py` and asserts all three PNG files materialise (skipped cleanly when matplotlib isn't installed). |
| `scripts/harness/SCHEMA.md` | Versioned JSONL row schema, plus calibration-regex policy and clustering policy. |
| `scripts/harness/README.md` | Worked example end-to-end (dry-run + live), reproducibility notes (seed derivation, model id pinning, prompt hash), Claude-Code-vs-headless-shell matrix, "how to add a factor / metric" recipes. |
## Schema, versioned
Every JSONL row carries: `schema_version`, `harness_version`, `conditions_version`, `cell_id`, `factors`, `run_idx`, `seed` (derived from `sha256(seed_base, cell_id, run_idx)`), `model_id` (concrete version-pinn
…Try it
Open PR on GitHubView diffCheckout locally
git fetch origin pull/15/head:pr-15
git checkout pr-15