Vote for your favorite SkillMD. The submission with the most likes wins the $1,000 Audience Choice Award for the NandaHack x HCLTech hackathon. Voting is open through September 25.Vote now →
in reviewhumanPR #72Data Facts

maurathat: sic_facts — serialization-invariant content addressing (closes the provenance-laundering gap in byte-level CIDs)

TL;DR: cid_facts prevents name substitution, but not identity forking under semantically equivalent serialization. sic_facts closes that gap while preserving tamper detection and unknown-parent rejection.

Author

Status
In review
Opened on
Jul 6
Branch
hackathon/maurathat-serialization-invariant-datafacts

Description

The pitch.

**TL;DR:** `cid_facts` prevents name substitution, but not identity forking under semantically equivalent serialization. `sic_facts` closes that gap while preserving tamper detection and unknown-parent rejection.

## Problem 08 follow-on: serialization-invariant datafacts addressing

PR #35, #34, and #59 closed the *name*-substitution attack by making datafact URLs content-addressed. This PR closes the remaining attack that byte-level content addressing still permits: **encoding substitution / provenance laundering**.

Today, `cid_facts` binds a dataset to a byte digest derived from the payload (`checksum`) plus `json.dumps` of the metadata dict. That prevents simple name substitution, but a semantically equivalent re-encoding still receives a fresh address:

- integer → float normalization (`1` re-exported as `1.0`)
- JSON key reordering
- NFC → NFD Unicode normalization
- re-gzipped exports (new bytes, same parsed structure)
- other encoding-level changes covered by the canonicalizer

None of these alter the dataset's structure, but each mints a new `df://sha256-*` URL. As a result, quarantine flags, annotations, ACL references, and provenance parents can be silently severed from the original dataset identity.

This PR adds `sic_facts`, which addresses datasets by a canonical **structural** digest instead of raw bytes. Semantically equivalent re-encodings covered by the canonicalizer collapse to the same URL; real tampering still changes the address; and provenance parents cited through a re-encoded alias resolve back to the original parent.

## New scenario

Adds `serialization_invariance` plus three adversarial validators:

- `sic_launder_neutralized`
- `sic_tamper_still_detected`
- `sic_phantom_parent_rejected`

With `layers.datafacts: sic_facts`, all validators pass. Flip one YAML line back to the currently merged `cid_facts` plugin and `sic_launder_neutralized` fails with forked addresses in the trace — a concrete gap in main, not a strawman.

## Receipts (reproduced on latest main @ 9a4e6fb, 2026-07-05 PT)

```
nest run scenarios/serialization_invariance.yaml
```

`layers.datafacts: sic_facts` (as committed):

```
PASS sic_launder_neutralized - re-encoded republish collapsed onto df://sic-9c03867928c831ad327ac8c11a0cf979454a7a52314a694d9273dc97a3881b3c
PASS sic_tamper_still_detected - tamper produced distinct address df://sic-4e7cb2b4f849f1ccbab2d24f4fa9eb6ebfb5e0acb7441f7e9baf8b59e6444f36
PASS sic_phantom_parent_rejected - unpublished parent rejected
```

Same scenario with one line flipped to `layers.datafacts: cid_facts`:

```
FAIL sic_launder_neutralized - re-encoded copy minted a fresh address: ['df://sha256-e6ebb4ffc8c8b5f4339c045148da0dc637fb1f51a4a6a4bb0115521b7ab96709', 'df://sha256-fe08b85307842ed986c9f2cd003c1f40c8110668ac7d83166b377cbd93d2eeb7'] (history severed)
PASS sic_tamper_still_detected - tamper produced distinct address df://sha256-33c6d2b6655a746696b7d5c5bb1acdbd79fb16d8a11f683e3f79c5f25bb6f70e
PASS sic_phantom_parent_rejected - unpublished parent rejected
```

## Test coverage

13 new tests, including an explicit contrast pair against `cid_facts`. All Problem 08 base criteria remain intact:

- content-addressed URLs
- provenance DAG with unknown-parent rejection
- identity-signed freshness proofs (logical ticks, never wall-clock)
- hash-keyed ACLs

## Local CI

```
make ci-local: green end-to-end
pytest: 749 passed / 0 failed
ruff check: clean
ruff format --check: clean
pyright: 0 errors
```

Deterministic throughout: logical ticks, seeded scenario, no wall-clock dependence, stdlib-only canonicalization.

## Background

The canonicalization approach follows my work on serialization-invariant content addressing for ML model artifacts (UOR-ADDR, UOR Foundation), where byte divergence across encoders for logically identical artifacts is the central failure mode.

---

This preserves all Problem-08 guarantees while adding serialization-invariant identity for canonicalized dataset structures.

Try it

Open PR on GitHubView diff

Checkout locally

git fetch origin pull/72/head:pr-72
git checkout pr-72