google-staff: netem transport + tail-latency metrics + SLO validators

Description

The pitch.

## Piece picked

**Layer 1 — Transport.** A `netem`-style alternative transport plugin, plus the observability and assertion machinery a real SRE would need to use it: tail-latency metrics and per-scenario SLO validators.

## Why

The README is honest about it:

> The default transport is zero-latency. ... `mean_latency` / `duration` will both be `0.0` in your trace. Latency *numbers* become meaningful only when ... you write a transport plugin that introduces per-hop delay.

The mean is the least interesting number in a distributed system anyway. **Real protocols fail at the tail.** Without a way to inject realistic per-link delay, jitter, bandwidth, and reorder, and without p50/p95/p99 metrics and pass/fail SLO checks, NEST can verify *correctness* but cannot answer "will this still satisfy 250ms p99 at 1000 agents?" — which is the question every protocol team eventually has to answer in production.

## Core idea

1. **`DelayModel`** (`nest_core.sim.delay_model`) — a netem-style per-link emulator: constant / uniform / lognormal-tailed latencies, jitter, bandwidth-based serialization delay, per-link drops, reproducible per-link FIFO. Pure with respect to a caller-supplied `random.Random`, so the simulator stays byte-deterministic.
2. **Simulator wiring** — `InMemoryTransport` accepts an optional `delay_model`. When `None` (the default), behaviour is byte-identical to today. When attached, every `send`/`broadcast` consults the model and the delivery event is scheduled at `now + delay`. Drops surface as `dropped` trace events with `reason: netem`.
3. **`netem` plugin** — `nest_plugins_reference.transport.netem` exposes a `StandaloneNetemTransport` for Tier-2 / shell use (delays via `asyncio.sleep`) and a `make_delay_model` helper. Registered as a built-in `("transport", "netem")` in the plugin registry.
4. **Scenario YAML** — new optional `transport:` block (parsed into `TransportConfig`) and new optional `slo:` block. The runner builds a `DelayModel` only when `layers.transport == "netem"`, seeded from `scenario.seed XOR transport.seed_salt` so reruns are reproducible and the transport's randomness can be perturbed independently of agent behaviour.
5. **Metrics** — added `p50_latency`, `p95_latency`, `p99_latency`, `max_latency`, computed from corr-paired send/receive events using nearest-rank (matches what netem and most prod tooling report).
6. **SLO validators** — `validate_latency_slo` checks `p50/p95/p99/max_latency` budgets and `min_delivery_rate`. Wired into the runner: scenarios with an `slo:` block get pass/fail results on `runner.validations` automatically.

## How to test

```bash
uv sync
uv run ruff check . && uv run ruff format --check .
uv run pyright
uv run pytest -q
```

All green on my machine: **293 tests pass** (up from 259 baseline; +34 new), zero ruff findings, zero pyright errors.

Try the bundled example:

```bash
cp examples/netem-transport/marketplace-netem.yaml /tmp/
uv run nest run /tmp/marketplace-netem.yaml -o /tmp/netem.jsonl
```

Or programmatically:

```python
from nest_core.runner import ScenarioRunner
from nest_core.scenario import ScenarioConfig

cfg = ScenarioConfig.from_yaml("examples/netem-transport/marketplace-netem.yaml")
runner = ScenarioRunner(cfg)
await runner.run()
print(runner.metrics)       # p50/p95/p99/max all > 0
print(runner.validations)   # pass/fail per SLO budget
```

A representative run from this branch on the bundled example:

```
mean_latency      0.034125 s
p50_latency       0.021009 s     (target 0.020)
p95_latency       0.111038 s
p99_latency       0.189972 s     (target 0.200, budget 0.250 -> PASS)
max_latency       0.820940 s     (heavy tail from lognormal)
slo_p99_latency           PASS  observed 0.189972s vs budget 0.250000s over 1000 messages
slo_min_delivery_rate     PASS  observed 1.0000 vs target 0.9900
```

Tightening the budget to `p99_latency: 0.050` immediately flips to `FAIL`, which is exactly what you want from a regression signal.

## Key assumptions

- **Determinism is non-negotiable.** Same `seed` + same YAML → byte-identical trace. The delay model uses its own seeded RNG so swapping `seed_salt` perturbs only the transport.
- **Backwards-compatible by default.** No existing scenario, validator, or metric changes behaviour. `delay_model=None` is the same code path as before.
- **Unit-sized messages.** This is a netem-style model for agent-protocol messages, not a packet-level DPDK emulator. Serialization delay is `bytes * 8 / kbps`.
- **Tier 1 only for the wired plugin path.** `StandaloneNetemTransport` exists for Tier 2 / shell use but is not deterministic by design (it sleeps in real time).
- **Nearest-rank percentiles**, since that is what netem and most production telemetry report and it never returns a value that wasn't actually in the distribution.

## Persona

Google staff engineer (Spanner / Borg DNA): metrics before features, SLOs over feelings, deterministic reproducibility over clever heuristics, and tail latency is the only latency that matters.

## Future work

- Per-link bandwidth queues (model HOL blocking under burst load, not just per-message serialization).
- A `chaos:` scenario hook that flips `seed_salt` mid-run for chaos-style regression sweeps.
- Hook the HTML report to plot the per-message latency CDF and color rows by SLO violation.
- A `validate_latency_slo` variant that operates on a *rolling window* so long-tail spikes show up even when the overall p99 stays under budget.
- Extend the model to Byzantine link behaviour (per-pair partition timers) so it composes cleanly with `failures.network_partition`.
- A `nest dashboard` panel that surfaces SLO pass/fail next to the metrics it depends on.

https://claude.ai/code/session_01C5j2D4MgCkPgsjSCqBVpWW

---
_Generated by [Claude Code](https://claude.ai/code/session_01C5j2D4MgCkPgsjSCqBVpWW)_