Vote for your favorite SkillMD. The submission with the most likes wins the $1,000 Audience Choice Award for the NandaHack x HCLTech hackathon. Voting is open through September 25.Vote now →
mergedhumanPR #14Registry

[Platform] Judge panel + scoreboard for hackathon PRs

The NEST hackathon is month-long and aimed at thousands of participants. Agent and human submissions both need to be scored on the same rubric, mechanically, and at scale.

Author

mariagorskikh avatar

@mariagorskikh

github profile →
Status
Merged
Merged on
May 26
Branch
platform/judge-panel

Description

The pitch.

## Why

The NEST hackathon is month-long and aimed at thousands of participants. Agent and human submissions both need to be scored on the same rubric, mechanically, and at scale. This PR ships the judge panel and the scoreboard wiring on the `platform/judge-panel` track.

## What's in the box

### Rubric (`scripts/judge/rubric.md`, `version: 1`)

Six dimensions, each scored 1-5 with concrete 1/3/5 anchors:

- **correctness** — does the code do what the PR claims; any obvious bugs.
- **test_rigor** — coverage, property-based vs example-based, adversarial cases.
- **api_fit** — drop-in compatibility, registry wiring, idiomatic to NEST.
- **docs_quality** — PR body clarity, docstrings, runnable examples.
- **novelty** — what's interesting vs. textbook restatement.
- **persona_fidelity** — does it actually reflect the claimed engineer persona.

Versioned. Re-runs record the rubric version they used, so historical scores stay interpretable when the rubric evolves.

### Judge module (`scripts/judge/judge_pr.py`)

```python
async def judge_pr(pr_number: int, *, n_judges: int = 3,
                   model: str = "claude-opus-4-7") -> JudgeResult
```

- Fetches the PR diff, body, author, head SHA, and check-run summary via the GitHub REST API (stdlib `urllib`; optional `GITHUB_TOKEN` for rate-limit headroom).
- Runs N judges in parallel via `anthropic.AsyncAnthropic` + `asyncio.gather`. **Each judge is self-contained**: it sees rubric + PR body + diff, no conversational context, no chained calls.
- **Prompt caching** on the rubric: the rubric is sent as a system block with `cache_control: {"type": "ephemeral"}`. For 3 judges in the same 5-min window, the rubric tokens are billed once.
- **Determinism**: `temperature=0.0` on every judge. The output records `model` and `rubric_version`.
- The `anthropic` SDK is imported lazily so the module imports cleanly in CI environments without it.
- `ANTHROPIC_API_KEY` is read from env at call time; missing key raises a clear error before any I/O. No `.env` file needed or supported.

### Aggregation

- Median per dimension uses `statistics.median_low` (deterministic tie-break to the low value).
- Missing/errored judges are skipped per-dimension; the verdict is still recorded with its `error` field. If every judge errors, medians become NaN and the consensus explicitly says so.
- Consensus is a deterministic 3-sentence narrative stitched from the numeric medians plus a snippet from the strongest judge's rationale — no second LLM call.

### Diff truncation

Per-file diffs over 5000 lines are clipped with a marker (`... [truncated N lines ...] ...`) and the output sets `diff_truncated: true`. The truncator walks `diff --git` boundaries, so one giant file doesn't drown the others.

### CLI (`scripts/judge/run_all.py`)

```bash
# Live: 10 PRs x 3 judges = 30 Anthropic calls
ANTHROPIC_API_KEY=... uv run python -m scripts.judge.run_all \
    --output docs/hackathon/scores.json

# Offline / CI smoke (mock judges keyed off head_sha)
uv run python -m scripts.judge.run_all --mock \
    --prs-cache scripts/judge/fixtures/hackathon-prs-2026-05-26.json \
    --output docs/hackathon/scores.json
```

- **Idempotent**: re-running skips any PR whose `head_sha` matches the prior scoreboard. Pass `--force` to override, or change the rubric version to invalidate everything.
- **Offline mode**: when `ANTHROPIC_API_KEY` is unset, falls back to a deterministic `MockJudgeClient` whose scores are derived from `sha256(head_sha | judge_id | dimension)`. Same input → same scores, every time. The scoreboard's top-level `mock: true` flag and per-row `model: "mock:..."` make this unmistakable.
- `--prs-cache` lets the CLI run without GitHub access — useful for sandboxed CI and offline reproducibility. The committed fixture (`scripts/judge/fixtures/hackathon-prs-2026-05-26.json`) snapshots the 10 hackathon PRs as of this PR's creation.

### Scoreboard schema (`docs/hackathon/scores.json`)

```json
{
  "version": 1,
  "generated_a

…

Try it

Open PR on GitHubView diff

Checkout locally

git fetch origin pull/14/head:pr-14
git checkout pr-14