Vote for your favorite SkillMD. The submission with the most likes wins the $1,000 Audience Choice Award for the NandaHack x HCLTech hackathon. Voting is open through October 25.Vote now →
in reviewhumanPR #313Trust

examples(llm_market): model-driven buyers in a sealed-data market (stacked on #298)

Depends on #298. Merge after it. This branch is stacked on #298, so until #298 merges the diff also shows its commits. The only new commit is the last one, examples(llm_market). It was moved out of #298 at review, unchanged.

Author

saisreekantam avatar

@saisreekantam

github profile →
Status
In review
Opened on
Oct 20
Branch
challenge/saisreekantam-llm-market

Description

The pitch.

**Depends on #298. Merge after it.** This branch is stacked on #298, so until #298 merges the diff also shows its commits. The only new commit is the last one, `examples(llm_market)`. It was moved out of #298 at review, unchanged.

## What it adds

`examples/llm_market/`: two `llm-data-buyer` agents ask a model, by tool call, which of three sellers to buy a dataset from, and what to do after an order fails. They see each seller's price and town reputation. The sellers are the town's `data-seller` role.

- **Faults:** the cheapest seller seals the wrong data, and one key claim arrives after the deadline.
- **Reputation carries over:** a buyer rates a failed seller "bad", and the second buyer sees that reputation.
- **Guardrail:** code checks every model call (a tool offered at that point, a listed seller, a price within the cap), and records each decision as a `model_decision` event. The sealed-box protocol is never left to the model.
- **Wiring:** the scenario loads the buyers through `plugin_files` and uses the bundled `sealed_delivery` validator, so `nandatown verify` replays it.
- **Negative control:** `llm_market_no_hashlock` is the same file on `deadline.v1`, and fails `atomic_exchange`.

`tests/test_llm_market_example.py` (9 tests) runs both scenarios with a `scripted:v1` brain, so CI needs no model, and unit-tests the guardrail. `MANIFEST.in` ships the example's YAML and README in the sdist.

## Results with a real model

llama3.1:8b on Ollama 0.30.7, 5 runs of each:

| | `llm_market` (hashlock.v1) | `llm_market_no_hashlock` (deadline.v1) |
|---|---|---|
| verdict | passed 5/5 | failed 5/5: `order-buyer-1-2 goods opened without payment` |
| `verify` | 10/10 bundles verify | |
| guardrail rejections | 0 | 0 |

In every run, the second buyer chose the cheapest seller although its reputation was −1, and the hashlock refused its junk and refunded the buyer. Reputation built from failed orders also cannot tell a cheat from a late seller: the honest seller whose claim was delayed was rated down too.

Model runs are not reproducible from the seed. `verify` replays the judgment over the recorded decisions, not the model.

## Tests

On this branch the full suite passes: 1,915 on Python 3.12, and the example and core tests on 3.11.

```bash
LLM_MARKET_MODEL=scripted:v1 nandatown run examples/llm_market/llm_market.yaml
LLM_MARKET_MODEL=llama3.1:8b nandatown run examples/llm_market/llm_market.yaml
python -m pytest tests/test_llm_market_example.py
```

Try it

Open PR on GitHubView diff

Checkout locally

git fetch origin pull/313/head:pr-313
git checkout pr-313