Delta Manifest Generation › Serializing Deterministic Discrepancy Manifests

Serializing Deterministic Discrepancy Manifests

This page answers one exacting question: how do you serialize a reconciliation manifest so that the same discrepancies always produce the same bytes, on any host, in any Python version, forever? It sits under the delta manifest generation reference and is the difference between a manifest digest that means something and one that changes for reasons that have nothing to do with the data. Byte-level determinism is what lets you compare two manifests by hash, sign one and trust the signature later, and prove to an auditor that a re-run reconstructed exactly the same verdict.

Problem Framing

You hash each manifest and store the digest as proof of a reconciliation run. A week later the same inputs produce a different digest, and every downstream assumption collapses. The causes are always the same small set of nondeterminism sources: dictionary or set iteration order, floating-point values formatted differently (0.1 versus 1e-1 versus 0.10000000000000001), unicode strings in different normalization forms that compare equal but serialize to different bytes, NaN and negative zero, and non-UTF-8 or locale-dependent encoding. Each is individually small; together they make an unsigned, casually serialized manifest worthless as an audit artifact. The fix is a single canonicalization pass applied before serialization.

Implementation

The canonicalizer normalizes every value to a stable form — sorted keys, fixed float representation, NFC unicode, explicit null handling — then serializes with a deterministic encoder. The output is pure ASCII bytes so transport and storage cannot reintroduce variance.

python
import json
import logging
import math
import unicodedata
from decimal import Decimal
from typing import Any

logging.basicConfig(
    format="%(asctime)s | %(levelname)s | %(name)s | %(message)s", level=logging.INFO
)
logger = logging.getLogger("recon.diffing.canonical")


def canonicalize(value: Any) -> Any:
    """Normalize a value to a byte-stable form before serialization."""
    if isinstance(value, str):
        # NFC so visually identical strings serialize to identical bytes.
        return unicodedata.normalize("NFC", value)
    if isinstance(value, float):
        if math.isnan(value) or math.isinf(value):
            raise ValueError("non-finite float is not serializable deterministically")
        # Fixed-precision decimal string kills 0.1-vs-1e-1 and -0.0 ambiguity.
        return format(Decimal(value).normalize(), "f") if value else "0"
    if isinstance(value, dict):
        return {k: canonicalize(value[k]) for k in sorted(value)}
    if isinstance(value, (list, tuple)):
        return [canonicalize(v) for v in value]
    return value


def serialize_canonical(payload: Any) -> bytes:
    """Deterministic ASCII bytes: sorted keys, no whitespace drift, NFC + fixed floats."""
    normalized = canonicalize(payload)
    text = json.dumps(normalized, sort_keys=True, separators=(",", ":"), ensure_ascii=True)
    return text.encode("ascii")

Key Implementation Notes

  • Sort keys at every level, not just the top. A nested object with unsorted keys reintroduces order-dependence even when the envelope is sorted. canonicalize recurses so every dictionary — however deep — emits keys in a fixed order.
  • Format floats through a fixed decimal representation. Raw repr(float) is stable within one CPython build but is a landmine across versions and languages. Converting through Decimal(...).normalize() and formatting with "f" gives one canonical string and collapses -0.0 to 0, which is the single most common silent digest change.
  • Normalize unicode to NFC before hashing. Two strings that render identically but use composed versus decomposed code points are equal to a human and different to a hasher. NFC makes them one byte sequence, so a name with an accent does not flip the digest between a Mac and a Linux extractor.
  • Reject non-finite values loudly. NaN, Infinity, and -Infinity have no canonical JSON form; serializing them silently produces invalid or divergent output. Fail fast so the upstream diff, not the manifest, is fixed. This discipline mirrors the deterministic hashing used in column-level checksum generation.

Verification

The harness is a byte-equality assertion: two independent serializations of logically equal payloads, built in different key orders and unicode forms, must produce identical bytes.

python
import hashlib

a = {"z": 1, "name": "café", "amount": 0.10, "flag": -0.0}
# Same data, different insertion order and a decomposed 'é' (e + combining accent).
b = {"amount": 1e-1, "flag": 0.0, "name": "café", "z": 1}

sa, sb = serialize_canonical(a), serialize_canonical(b)
assert sa == sb, (sa, sb)
assert hashlib.sha256(sa).hexdigest() == hashlib.sha256(sb).hexdigest()

# Non-finite floats must be rejected, not silently serialized.
try:
    serialize_canonical({"x": float("nan")})
    raise AssertionError("NaN should have been rejected")
except ValueError:
    pass
logger.info("deterministic serialization harness passed")

Wire this as a build gate: run it in CI so any change to the serializer, a dependency bump, or a new field type that reintroduces nondeterminism fails the build before it can corrupt a stored digest.

Operational Considerations

Pin the serializer and its dependencies, and run the byte-equality harness on every deploy across the actual Python versions your extractors use, because a determinism guarantee that holds only on the build host is no guarantee at all. Keep volatile metadata — run id, wall clock, hostname — out of the hashed body and in an unhashed sidecar, so a manifest’s digest reflects only the data. Store manifests as the ASCII bytes the serializer emits, never as re-encoded or pretty-printed copies, since any reformatting breaks the digest. When the manifest feeds a signed audit record, hash with a NIST-approved algorithm so the fingerprint is both reproducible and defensible in a regulated review.