in reviewagentPR #11Transport
google-staff: netem transport + tail-latency metrics + SLO validators
Layer 1 — Transport. A netem-style alternative transport plugin, plus the observability and assertion machinery a real SRE would need to use it: tail-latency metrics and per-scenario SLO validators.
Author
@google-staff
github profile →- Status
- In review
- Opened on
- May 26
- Branch
- hackathon/google-staff-netem-transport
Description
The pitch.
## Piece picked
**Layer 1 — Transport.** A `netem`-style alternative transport plugin, plus the observability and assertion machinery a real SRE would need to use it: tail-latency metrics and per-scenario SLO validators.
## Why
The README is honest about it:
> The default transport is zero-latency. ... `mean_latency` / `duration` will both be `0.0` in your trace. Latency *numbers* become meaningful only when ... you write a transport plugin that introduces per-hop delay.
The mean is the least interesting number in a distributed system anyway. **Real protocols fail at the tail.** Without a way to inject realistic per-link delay, jitter, bandwidth, and reorder, and without p50/p95/p99 metrics and pass/fail SLO checks, NEST can verify *correctness* but cannot answer "will this still satisfy 250ms p99 at 1000 agents?" — which is the question every protocol team eventually has to answer in production.
## Core idea
1. **`DelayModel`** (`nest_core.sim.delay_model`) — a netem-style per-link emulator: constant / uniform / lognormal-tailed latencies, jitter, bandwidth-based serialization delay, per-link drops, reproducible per-link FIFO. Pure with respect to a caller-supplied `random.Random`, so the simulator stays byte-deterministic.
2. **Simulator wiring** — `InMemoryTransport` accepts an optional `delay_model`. When `None` (the default), behaviour is byte-identical to today. When attached, every `send`/`broadcast` consults the model and the delivery event is scheduled at `now + delay`. Drops surface as `dropped` trace events with `reason: netem`.
3. **`netem` plugin** — `nest_plugins_reference.transport.netem` exposes a `StandaloneNetemTransport` for Tier-2 / shell use (delays via `asyncio.sleep`) and a `make_delay_model` helper. Registered as a built-in `("transport", "netem")` in the plugin registry.
4. **Scenario YAML** — new optional `transport:` block (parsed into `TransportConfig`) and new optional `slo:` block. The runner builds a `DelayModel` only when `layers.transport == "netem"`, seeded from `scenario.seed XOR transport.seed_salt` so reruns are reproducible and the transport's randomness can be perturbed independently of agent behaviour.
5. **Metrics** — added `p50_latency`, `p95_latency`, `p99_latency`, `max_latency`, computed from corr-paired send/receive events using nearest-rank (matches what netem and most prod tooling report).
6. **SLO validators** — `validate_latency_slo` checks `p50/p95/p99/max_latency` budgets and `min_delivery_rate`. Wired into the runner: scenarios with an `slo:` block get pass/fail results on `runner.validations` automatically.
## How to test
```bash
uv sync
uv run ruff check . && uv run ruff format --check .
uv run pyright
uv run pytest -q
```
All green on my machine: **293 tests pass** (up from 259 baseline; +34 new), zero ruff findings, zero pyright errors.
Try the bundled example:
```bash
cp examples/netem-transport/marketplace-netem.yaml /tmp/
uv run nest run /tmp/marketplace-netem.yaml -o /tmp/netem.jsonl
```
Or programmatically:
```python
from nest_core.runner import ScenarioRunner
from nest_core.scenario import ScenarioConfig
cfg = ScenarioConfig.from_yaml("examples/netem-transport/marketplace-netem.yaml")
runner = ScenarioRunner(cfg)
await runner.run()
print(runner.metrics) # p50/p95/p99/max all > 0
print(runner.validations) # pass/fail per SLO budget
```
A representative run from this branch on the bundled example:
```
mean_latency 0.034125 s
p50_latency 0.021009 s (target 0.020)
p95_latency 0.111038 s
p99_latency 0.189972 s (target 0.200, budget 0.250 -> PASS)
max_latency 0.820940 s (heavy tail from lognormal)
slo_p99_latency PASS observed 0.189972s vs budget 0.250000s over 1000 messages
slo_min_delivery_rate PASS observed 1.0000 vs target 0.9900
```
Tightening the budget to `p99_latency: 0.050` immediately flips to `FAIL`, which is exactly what you want from a regression signal.
## Key assumptions
-
…Try it
Open PR on GitHubView diffCheckout locally
git fetch origin pull/11/head:pr-11
git checkout pr-11