Vote for your favorite SkillMD. The submission with the most likes wins the $1,000 Audience Choice Award for the NandaHack x HCLTech hackathon. Voting is open through October 25.Vote now →
in reviewhumanPR #293Identity

[Fellowship Challenge] Track: a poison request is redelivered forever; claim + reproduction

Submission for the Hexaware x Project NANDA Fellowship Technical Challenge

Author

priyansh-saxena1 avatar

@priyansh-saxena1

github profile →
Status
In review
Opened on
Sep 30
Branch
challenge/priyansh-saxena1-dead-letter

Description

The pitch.

Submission for the Hexaware x Project NANDA Fellowship Technical Challenge

> The first commit reproduces the bug (its test fails on main). The fix, the profile, the negative control and the evaluator stages are on top, and everything passes and verifies.

| | |
|---|---|
| **Track** | Track (the coordinator's mailbox) |
| **Gap** | Work a consumer can never finish is handed out forever and the sender is never told. |
| **Fault** | A poison request: the consumer fails every delivery of it. |
| **Invariant** | A request gets at most `max_attempts` deliveries; if it can't be done the town dead-letters it once and tells the sender once. |
| **Lands in** | Track profile `quote-poison-request` (`max_attempts: 3`) |
| **Negative control** | `quote-poison-unbounded`, same recipe with no budget; FAILS at `custody_ended`, as predicted |
| **Bundle fingerprint** | `sha256:73f137994a3d18eeb144fc6dc5df97a353a16809c0ac17dac918f06869822072` (PASSED, verify ok) |

*Negative control fingerprint:* `sha256:f491e9fb1a938c0f5f725a3d077b949ba305a40b9eb02b8d4e94116bc9204052` (FAILED at custody_ended, verify ok)

## The problem

So basically if a seller cant handle a request then the town just keeps giving the same request back to it. Over and over. Nothing stops it wasting resources and  buyer never hears anything, the run just ends up timing out as INCOMPLETE.

Here's what I actually saw:
- A seller that always answers "retry later" got the same request **319 times in 45 seconds**.
- A seller that said "failed" just once still left the buyer sitting there for the full 45 seconds.

code issue: `TownDB.claim_next` counts the attempts but never limits them and a `retryable` ack just puts the work back in the inbox.

## Why this one

- The Track is about custody of work and right now custody can start but it never ends.
- It would turn a vague timeout into something you can actually point at.
- Real queues (like kafka, rabbitmq) limit retries and put failed work aside in a "dead letter" place. AWS guidance on reliable agents also lists dead letter handling as a must so I think this is high priority fix.
- Nowhere proposed --> went through all 282 PRs.

## Reproduce

**Tests** (seconds, no network):

```bash
python -m pip install -e ".[dev]"
python -m pytest tests/test_poison_request.py -q
```

On main all 6 cases fail, like this:

```
q-1 was delivered 8 times (attempts [1, .., 8]) and was still being handed out when the consumer stopped asking now the profile allows 3 expected one message_dead_lettered event for q-1 but in fact there were 0: the output does show that custody of q- 1 ended the buyer inbox holds nothing about q-1 so it can only wait until its deadline
```

**End to end** (about 60 seconds): a seller that always answers "retry later".

```bash
cat > /tmp/poison_seller.py <<'PY'
import json, os, time, urllib.request
T, R, N, K = (os.environ[k] for k in ("TOWN_URL", "RUN_ID", "NAME", "TOKEN"))
deadline = time.time() + float(os.environ.get("DEADLINE", "45"))
session = None
def call(method, path, body=None):
    req = urllib.request.Request(f"{T}/runs/{R}{path}", method=method,
                                 data=json.dumps(body).encode() if body is not None else None)
    req.add_header("Content-Type", "application/json")
    if session:
        req.add_header("X-Town-Session", session)
    with urllib.request.urlopen(req, timeout=10) as resp:
        return None if resp.status == 204 else json.loads(resp.read() or b"{}")
session = call("POST", "/join", {"name": N, "token": K})["session"]
while time.time() < deadline:
    claim = call("POST", "/inbox/claim")
    if claim is None:
        time.sleep(0.1)
        continue
    call("POST", "/inbox/ack", {"message_id": claim["message_id"], "fence": claim["fence"],
                                "status": "retryable", "note": {"reason": "cannot parse"}})
PY
nandatown test-agent --role seller --cmd "python /tmp/poison_se

…

Try it

Open PR on GitHubView diff

Checkout locally

git fetch origin pull/293/head:pr-293
git checkout pr-293