Vote for your favorite SkillMD. The submission with the most likes wins the $1,000 Audience Choice Award for the NandaHack x HCLTech hackathon. Voting is open through October 25.Vote now →
in reviewhumanPR #294Coordination

[Fellowship Challenge] Partition fault: consensus stays safe while the town is split

Submission for the Hexaware x Project NANDA Fellowship Technical Challenge

Author

Paramjeet-singh-neu avatar

@Paramjeet-singh-neu

github profile →
Status
In review
Opened on
Sep 30
Branch
challenge/Paramjeet-singh-neu-consensus-partition

Description

The pitch.

Submission for the Hexaware x Project NANDA Fellowship Technical Challenge

**Gap**
The Lab transport can drop, duplicate or delay a single message, but it can't split the town so two groups of agents can't reach each other. Imported scenarios that declare `network_partition` just get reported as not modeled.

**Layer**
Transport (Lab), exercised by a new scenario, `consensus_partition`, and its control, `consensus_partition_quorum_defect`.

**Fault**
`action: partition` with `groups`, `start` and `heal`. While the cut lasts, anything sent between groups is dropped by the town, recorded as `message_dropped` with fault `partition` and the sender. After the heal, new messages go through and the proposer's normal retries take over. A message is judged when it's sent. The declaration fails closed: every agent in exactly one group, start before heal, heal before `max_time`, and for now a partition has to be the only fault in its scenario.

**Invariant**
The proposer can't commit without a majority of the configured acceptors, it does commit after the heal, and every acceptor ends up with the same value.

**Negative control**
The same partition, with a proposer that shrinks its quorum to whoever answered (`defect: shrink_quorum_on_timeout`, off unless named; unknown names raise). On a healthy network it never shows up.

**Predictions vs results**
I wrote the predictions into this PR before running either scenario.

| Run | Stage | Predicted | Result |
|---|---|---|---|
| `consensus_partition` | `partition_enforced` | pass | pass: 14 drops, none delivered |
| | `quorum_commit` | pass | pass |
| | `agreement` | pass | pass |
| | `progress_after_heal` | pass, commit around t=10.5–11.5 | pass, commit at t=11.2 |
| quorum defect | `partition_enforced` | pass | pass: 4 drops |
| | `quorum_commit` | fail, 1 of 3 acceptors | fail |
| | `agreement` | fail | fail |
| | `progress_after_heal` | fail | fail |

**Why two new stages**
One test makes the partition leak on purpose. `quorum_commit` and `agreement` still pass on that run, while `partition_enforced` and `progress_after_heal` fail. The existing checks can't tell a working partition from a leaky one.

**Evidence**
- Bundle fingerprints (seed 42):
  - `consensus_partition`: `sha256:3d5634df7cdda4482d143aee7cceaf21ba2dac48b4371eaf7e8cf7f8c96acd12`
  - `consensus_partition_quorum_defect`: `sha256:a08ee505ce7bb97d9ee423ed1d3c7deb212bf4d3c13caf9bae23683234d18efc`
- `nandatown verify` passes on both.
- Campaigns, 20 trials each: 20/20 passed, and 20/20 failed with the same stage pattern every time. The scenario makes no random choices, so this shows the result doesn't depend on the seed. It isn't a reliability claim.
- The 7 existing scenarios give identical events and verdicts at seed 42 compared with `main@36e7f6c` (run IDs and evaluation time aside), so `LAB_EVALUATOR_VERSION` stays at lab-0.2.6.
- Full suite: 1801 passed. The new tests also pass on Python 3.11, and the wheel ships both scenarios and runs them outside the checkout.

**What this doesn't prove**
- There's a single proposer, so two conflicting values can't happen. This tests the majority rule, not full split-brain protection.
- Lookup is an in-process call, so a cut-off agent can still discover agents it can't reach.
- It's Lab only, on one machine, not a real network.
- A partition can't be combined with other faults yet, because the order relative to per-message counters isn't defined.

**What's next**
- Partitions between real processes in the Track. While looking at that, I found the coordinator is the only path for recording evidence: kill it mid-run and no bundle is written at all.
- Map imported `network_partition` declarations onto this fault instead of reporting them as not modeled.
- Re-test partition-tolerant protocols like legacy #33, now that the town can cut the network.

**Commits**
1. Declare timed partition faults, failing closed, with round-trip tests
2.

…

Try it

Open PR on GitHubView diff

Checkout locally

git fetch origin pull/294/head:pr-294
git checkout pr-294