mergedhumanPR #14Registry
[Platform] Judge panel + scoreboard for hackathon PRs
The NEST hackathon is month-long and aimed at thousands of participants. Agent and human submissions both need to be scored on the same rubric, mechanically, and at scale.
Description
The pitch.
## Why
The NEST hackathon is month-long and aimed at thousands of participants. Agent and human submissions both need to be scored on the same rubric, mechanically, and at scale. This PR ships the judge panel and the scoreboard wiring on the `platform/judge-panel` track.
## What's in the box
### Rubric (`scripts/judge/rubric.md`, `version: 1`)
Six dimensions, each scored 1-5 with concrete 1/3/5 anchors:
- **correctness** — does the code do what the PR claims; any obvious bugs.
- **test_rigor** — coverage, property-based vs example-based, adversarial cases.
- **api_fit** — drop-in compatibility, registry wiring, idiomatic to NEST.
- **docs_quality** — PR body clarity, docstrings, runnable examples.
- **novelty** — what's interesting vs. textbook restatement.
- **persona_fidelity** — does it actually reflect the claimed engineer persona.
Versioned. Re-runs record the rubric version they used, so historical scores stay interpretable when the rubric evolves.
### Judge module (`scripts/judge/judge_pr.py`)
```python
async def judge_pr(pr_number: int, *, n_judges: int = 3,
model: str = "claude-opus-4-7") -> JudgeResult
```
- Fetches the PR diff, body, author, head SHA, and check-run summary via the GitHub REST API (stdlib `urllib`; optional `GITHUB_TOKEN` for rate-limit headroom).
- Runs N judges in parallel via `anthropic.AsyncAnthropic` + `asyncio.gather`. **Each judge is self-contained**: it sees rubric + PR body + diff, no conversational context, no chained calls.
- **Prompt caching** on the rubric: the rubric is sent as a system block with `cache_control: {"type": "ephemeral"}`. For 3 judges in the same 5-min window, the rubric tokens are billed once.
- **Determinism**: `temperature=0.0` on every judge. The output records `model` and `rubric_version`.
- The `anthropic` SDK is imported lazily so the module imports cleanly in CI environments without it.
- `ANTHROPIC_API_KEY` is read from env at call time; missing key raises a clear error before any I/O. No `.env` file needed or supported.
### Aggregation
- Median per dimension uses `statistics.median_low` (deterministic tie-break to the low value).
- Missing/errored judges are skipped per-dimension; the verdict is still recorded with its `error` field. If every judge errors, medians become NaN and the consensus explicitly says so.
- Consensus is a deterministic 3-sentence narrative stitched from the numeric medians plus a snippet from the strongest judge's rationale — no second LLM call.
### Diff truncation
Per-file diffs over 5000 lines are clipped with a marker (`... [truncated N lines ...] ...`) and the output sets `diff_truncated: true`. The truncator walks `diff --git` boundaries, so one giant file doesn't drown the others.
### CLI (`scripts/judge/run_all.py`)
```bash
# Live: 10 PRs x 3 judges = 30 Anthropic calls
ANTHROPIC_API_KEY=... uv run python -m scripts.judge.run_all \
--output docs/hackathon/scores.json
# Offline / CI smoke (mock judges keyed off head_sha)
uv run python -m scripts.judge.run_all --mock \
--prs-cache scripts/judge/fixtures/hackathon-prs-2026-05-26.json \
--output docs/hackathon/scores.json
```
- **Idempotent**: re-running skips any PR whose `head_sha` matches the prior scoreboard. Pass `--force` to override, or change the rubric version to invalidate everything.
- **Offline mode**: when `ANTHROPIC_API_KEY` is unset, falls back to a deterministic `MockJudgeClient` whose scores are derived from `sha256(head_sha | judge_id | dimension)`. Same input → same scores, every time. The scoreboard's top-level `mock: true` flag and per-row `model: "mock:..."` make this unmistakable.
- `--prs-cache` lets the CLI run without GitHub access — useful for sandboxed CI and offline reproducibility. The committed fixture (`scripts/judge/fixtures/hackathon-prs-2026-05-26.json`) snapshots the 10 hackathon PRs as of this PR's creation.
### Scoreboard schema (`docs/hackathon/scores.json`)
```json
{
"version": 1,
"generated_a
…Try it
Open PR on GitHubView diffCheckout locally
git fetch origin pull/14/head:pr-14
git checkout pr-14