in reviewhumanPR #313Trust
examples(llm_market): model-driven buyers in a sealed-data market (stacked on #298)
Depends on #298. Merge after it. This branch is stacked on #298, so until #298 merges the diff also shows its commits. The only new commit is the last one, examples(llm_market). It was moved out of #298 at review, unchanged.
Author
@saisreekantam
github profile →- Status
- In review
- Opened on
- Oct 20
- Branch
- challenge/saisreekantam-llm-market
Description
The pitch.
**Depends on #298. Merge after it.** This branch is stacked on #298, so until #298 merges the diff also shows its commits. The only new commit is the last one, `examples(llm_market)`. It was moved out of #298 at review, unchanged. ## What it adds `examples/llm_market/`: two `llm-data-buyer` agents ask a model, by tool call, which of three sellers to buy a dataset from, and what to do after an order fails. They see each seller's price and town reputation. The sellers are the town's `data-seller` role. - **Faults:** the cheapest seller seals the wrong data, and one key claim arrives after the deadline. - **Reputation carries over:** a buyer rates a failed seller "bad", and the second buyer sees that reputation. - **Guardrail:** code checks every model call (a tool offered at that point, a listed seller, a price within the cap), and records each decision as a `model_decision` event. The sealed-box protocol is never left to the model. - **Wiring:** the scenario loads the buyers through `plugin_files` and uses the bundled `sealed_delivery` validator, so `nandatown verify` replays it. - **Negative control:** `llm_market_no_hashlock` is the same file on `deadline.v1`, and fails `atomic_exchange`. `tests/test_llm_market_example.py` (9 tests) runs both scenarios with a `scripted:v1` brain, so CI needs no model, and unit-tests the guardrail. `MANIFEST.in` ships the example's YAML and README in the sdist. ## Results with a real model llama3.1:8b on Ollama 0.30.7, 5 runs of each: | | `llm_market` (hashlock.v1) | `llm_market_no_hashlock` (deadline.v1) | |---|---|---| | verdict | passed 5/5 | failed 5/5: `order-buyer-1-2 goods opened without payment` | | `verify` | 10/10 bundles verify | | | guardrail rejections | 0 | 0 | In every run, the second buyer chose the cheapest seller although its reputation was −1, and the hashlock refused its junk and refunded the buyer. Reputation built from failed orders also cannot tell a cheat from a late seller: the honest seller whose claim was delayed was rated down too. Model runs are not reproducible from the seed. `verify` replays the judgment over the recorded decisions, not the model. ## Tests On this branch the full suite passes: 1,915 on Python 3.12, and the example and core tests on 3.11. ```bash LLM_MARKET_MODEL=scripted:v1 nandatown run examples/llm_market/llm_market.yaml LLM_MARKET_MODEL=llama3.1:8b nandatown run examples/llm_market/llm_market.yaml python -m pytest tests/test_llm_market_example.py ```
Try it
Open PR on GitHubView diffCheckout locally
git fetch origin pull/313/head:pr-313
git checkout pr-313