in reviewhumanPR #72Data Facts
maurathat: sic_facts — serialization-invariant content addressing (closes the provenance-laundering gap in byte-level CIDs)
TL;DR: cid_facts prevents name substitution, but not identity forking under semantically equivalent serialization. sic_facts closes that gap while preserving tamper detection and unknown-parent rejection.
Author
@maurathat
github profile →- Status
- In review
- Opened on
- Jul 6
- Branch
- hackathon/maurathat-serialization-invariant-datafacts
Description
The pitch.
**TL;DR:** `cid_facts` prevents name substitution, but not identity forking under semantically equivalent serialization. `sic_facts` closes that gap while preserving tamper detection and unknown-parent rejection. ## Problem 08 follow-on: serialization-invariant datafacts addressing PR #35, #34, and #59 closed the *name*-substitution attack by making datafact URLs content-addressed. This PR closes the remaining attack that byte-level content addressing still permits: **encoding substitution / provenance laundering**. Today, `cid_facts` binds a dataset to a byte digest derived from the payload (`checksum`) plus `json.dumps` of the metadata dict. That prevents simple name substitution, but a semantically equivalent re-encoding still receives a fresh address: - integer → float normalization (`1` re-exported as `1.0`) - JSON key reordering - NFC → NFD Unicode normalization - re-gzipped exports (new bytes, same parsed structure) - other encoding-level changes covered by the canonicalizer None of these alter the dataset's structure, but each mints a new `df://sha256-*` URL. As a result, quarantine flags, annotations, ACL references, and provenance parents can be silently severed from the original dataset identity. This PR adds `sic_facts`, which addresses datasets by a canonical **structural** digest instead of raw bytes. Semantically equivalent re-encodings covered by the canonicalizer collapse to the same URL; real tampering still changes the address; and provenance parents cited through a re-encoded alias resolve back to the original parent. ## New scenario Adds `serialization_invariance` plus three adversarial validators: - `sic_launder_neutralized` - `sic_tamper_still_detected` - `sic_phantom_parent_rejected` With `layers.datafacts: sic_facts`, all validators pass. Flip one YAML line back to the currently merged `cid_facts` plugin and `sic_launder_neutralized` fails with forked addresses in the trace — a concrete gap in main, not a strawman. ## Receipts (reproduced on latest main @ 9a4e6fb, 2026-07-05 PT) ``` nest run scenarios/serialization_invariance.yaml ``` `layers.datafacts: sic_facts` (as committed): ``` PASS sic_launder_neutralized - re-encoded republish collapsed onto df://sic-9c03867928c831ad327ac8c11a0cf979454a7a52314a694d9273dc97a3881b3c PASS sic_tamper_still_detected - tamper produced distinct address df://sic-4e7cb2b4f849f1ccbab2d24f4fa9eb6ebfb5e0acb7441f7e9baf8b59e6444f36 PASS sic_phantom_parent_rejected - unpublished parent rejected ``` Same scenario with one line flipped to `layers.datafacts: cid_facts`: ``` FAIL sic_launder_neutralized - re-encoded copy minted a fresh address: ['df://sha256-e6ebb4ffc8c8b5f4339c045148da0dc637fb1f51a4a6a4bb0115521b7ab96709', 'df://sha256-fe08b85307842ed986c9f2cd003c1f40c8110668ac7d83166b377cbd93d2eeb7'] (history severed) PASS sic_tamper_still_detected - tamper produced distinct address df://sha256-33c6d2b6655a746696b7d5c5bb1acdbd79fb16d8a11f683e3f79c5f25bb6f70e PASS sic_phantom_parent_rejected - unpublished parent rejected ``` ## Test coverage 13 new tests, including an explicit contrast pair against `cid_facts`. All Problem 08 base criteria remain intact: - content-addressed URLs - provenance DAG with unknown-parent rejection - identity-signed freshness proofs (logical ticks, never wall-clock) - hash-keyed ACLs ## Local CI ``` make ci-local: green end-to-end pytest: 749 passed / 0 failed ruff check: clean ruff format --check: clean pyright: 0 errors ``` Deterministic throughout: logical ticks, seeded scenario, no wall-clock dependence, stdlib-only canonicalization. ## Background The canonicalization approach follows my work on serialization-invariant content addressing for ML model artifacts (UOR-ADDR, UOR Foundation), where byte divergence across encoders for logically identical artifacts is the central failure mode. --- This preserves all Problem-08 guarantees while adding serialization-invariant identity for canonicalized dataset structures.
Try it
Open PR on GitHubView diffCheckout locally
git fetch origin pull/72/head:pr-72
git checkout pr-72