Vote for your favorite SkillMD. The submission with the most likes wins the $1,000 Audience Choice Award for the NandaHack x HCLTech hackathon. Voting is open through September 25.Vote now →
mergedhumanPR #31Data Facts

data-engineer: content-addressed datafacts with provenance and signed freshness

[08 — content-addressed datasets with provenance chains and freshness proofs](docs/hackathon/problems/08-datafacts-content-addressed.md).

Author

Status
Merged
Merged on
Jul 1
Branch
hackathon/data-engineer-datafacts-cid

Description

The pitch.

## Problem

[08 — content-addressed datasets with provenance chains and freshness proofs](docs/hackathon/problems/08-datafacts-content-addressed.md).

`datafacts_v1` addresses datasets by the publisher's chosen *name*, so a republish under the same name silently overwrites whatever was there; "freshness" is `time.time() - timestamp < 3600` with no idea who touched it; and there's no lineage field at all.

## What this adds

`cid_facts` (`packages/nest-plugins-reference/.../datafacts/cid_facts.py`):

- **URL = `sha256` of the dataset's content fields** (owner, description, schema_version, tags, checksum, size_bytes, access_tier, metadata — not the `name` label, not timestamps). Same content → same address; different content can't collide on one. Republishing is idempotent.
- **Provenance via `metadata["parents"]`** — a list of parent URLs; `publish()` rejects any parent it hasn't itself seen. It's a DAG, not a chain: a dataset can declare multiple parents (a join), and parent order is normalized before hashing so `[A,B]` and `[B,A]` resolve to one address.
- **Freshness is a signed proof, not a clock read** — each publish signs `(url, tick)` with the identity layer's key over a logical tick (no wall-clock, so Tier 1 stays deterministic). `verify_freshness` only accepts a proof signed by the dataset's declared owner.

## The three attacks, and why a diamond

The brief calls out three holes — substitution, stale-claim, broken-provenance — and also that *"provenance is a DAG, not a tree."* So the scenario is a real **diamond**: `source → {refine-a, refine-b} → aggregate(join) → verify`. That fork is what gives the work its teeth:

- `aggregate-0` is a join — it lists *both* refiners as parents, so the report's lineage fans back out to a single shared root.
- `verify-0` walks the lineage with a full breadth-first traversal over *every* parent, not just `parents[0]`. A first-parent-only walk would silently miss half the graph; the diamond makes that walk load-bearing rather than decorative.
- Content-addressing makes the DAG **acyclic and tamper-evident for free**: a self-parent or cycle is impossible by construction (a node's address isn't known until its parents are fixed), and altering any ancestor re-addresses every descendant. These are proven in tests, not just asserted.

## Scenario & validators

`scenarios/provenance_supply_chain.yaml` runs the diamond, then `verify-0` (as an outsider, never the source) runs the three attacks. Four validators read the trace: `provenance_chain_integrity` (happy path) plus the three adversarial ones. Flip `layers.datafacts` between `cid_facts` and `datafacts_v1` in the same YAML and all three adversarial checks flip PASS↔FAIL — same scenario, same attacker logic, only the plugin underneath changes.

## Verification

```bash
uv sync
nest run scenarios/provenance_supply_chain.yaml
python -c "from pathlib import Path; from nest_core.validators import validate_trace; \
    [print('PASS' if r.passed else 'FAIL', r.name, '-', r.detail) \
     for r in validate_trace(Path('traces/provenance_supply_chain.jsonl'), 'provenance_supply_chain')]"
```

All four pass on `cid_facts`; swap to `datafacts_v1` and the three adversarial ones flip to FAIL.

```bash
make ci-local   # 585 passed
```

## Tests

- `test_cid_facts.py`: content-addressing (idempotent republish, distinct-content/distinct-URL, name has no effect); provenance (unknown-parent rejection, multi-hop walk, multi-parent join with order-independent addressing and partial-failure atomicity); DAG properties (diamond ancestor set counts a shared root once, self-reference impossible, no cycle can be formed, altering an ancestor re-addresses every descendant); freshness (happy path, window expiry, forged-claim rejection, owner re-sign stays fresh); ACL; registry wiring; a hypothesis property suite over `content_hash`.
- `test_provenance_supply_chain.py`: per-validator synthetic-trace unit tests, the diamond walk visiting all four nodes, end-to-end PASS

…

Try it

Open PR on GitHubView diff

Checkout locally

git fetch origin pull/31/head:pr-31
git checkout pr-31