closedhumanPR #39Data Facts
hiten: Layer 12 DataFacts — content-addressed provenance …
The existing DataFacts layer could store and retrieve datasets, but it could not reliably answer a critical trust question:
Author
@Hiten0305l
github profile →- Status
- Closed without merging
- Opened on
- Jun 25
- Branch
- hackathon/hiten-provenance-dag-audit
Description
The pitch.
## Problem The existing DataFacts layer could store and retrieve datasets, but it could not reliably answer a critical trust question: > Where did this data come from, and can its complete lineage be verified? Datasets were identified by publisher-chosen names, freshness relied on timestamps, and provenance relationships were limited. Agents could verify that data existed, but not reliably verify its origin, integrity, freshness, or derivation history. ## What This Adds ### cid_facts A content-addressed DataFacts implementation. URL = SHA-256 of semantic dataset content: owner, description, schema_version, tags, checksum, size_bytes, access_tier, metadata, and normalized parent URLs — not the dataset name and not timestamps. Same content resolves to the same address, different content resolves to a different address, and republishing identical content is idempotent. Tags are normalized before hashing, so equivalent tag sets resolve to the same address regardless of order. Parent URLs are also normalized before hashing, so `[A,B]` and `[B,A]` resolve to the same address. ### Provenance DAGs Provenance is modeled as a DAG rather than a simple chain. Datasets declare parents through `metadata["parents"]`. A dataset may reference multiple parents, enabling joins and derived datasets with multiple upstream sources. Publishing rejects unknown parent references, so lineage can only be built from datasets already known to the system. The supply-chain scenario was generalized into: ```text N suppliers → M manufacturers → distributor → retailer ``` allowing larger provenance graphs without code changes. ### Freshness Proofs Freshness is a signed proof rather than a timestamp check. Each publish operation signs: ```text (dataset_url, logical_tick) ``` using the publisher's identity key. Verification only accepts proofs signed by the dataset's declared owner. No wall-clock time is used, preserving deterministic Tier-1 execution. ### Full Lineage Verification Retailers verify the complete provenance graph rather than only immediate parents. Verification uses a full breadth-first traversal (BFS), visits every reachable ancestor, deduplicates shared ancestors, validates freshness proofs throughout the lineage, and detects broken provenance chains. A first-parent-only walk would miss part of the graph; the multi-parent supply-chain DAG makes full traversal load-bearing. ### Provenance Audit Events After successful verification, retailers emit: ```text provenance_audit:<url>:nodes=<count>:ancestors=<urls> ``` This gives the trace an explainable record of exactly what lineage was discovered and verified. ## Security & Validation The implementation targets the three main DataFacts failure modes: substitution, stale freshness claims, and broken provenance. Validator tests use synthetic traces to prove the difference between weak traces and content-addressed provenance traces: * name-based or substituted URLs fail substitution checks * traces without valid signed freshness proofs fail freshness validation * derived datasets with unpublished parents fail provenance-chain validation Provenance tests exercise the real supply-chain DAG: suppliers feed manufacturers, manufacturers feed the distributor, and the retailer verifies the final dataset by walking the full lineage. Shared ancestors are intentionally exercised so the traversal proves they are visited exactly once. Scenario/factory tests verify configurable topology: different supplier/manufacturer counts can be created from config, manufacturers wait for all required supplier URLs, distributors wait for all required manufacturer URLs, and retailers emit `provenance_audit` after successful verification. ## Verification ```bash uv sync uv run ruff check . uv run ruff format --check . uv run pyright uv run pytest -v ``` Results: ```text 58 passed in test_cid_facts.py 599 passed, 1 skipped, 1 deselected …
Try it
Open PR on GitHubView diffCheckout locally
git fetch origin pull/39/head:pr-39
git checkout pr-39