Structural Diffing & Sync Engines › Delta Manifest Generation

# Delta Manifest Generation

Delta manifest generation is the workload that turns a raw comparison result into the artifact everything downstream depends on: an ordered, deterministic, serializable record of every add, drop, and value divergence between two systems, annotated with the schema version, tolerance policy, and key range that produced it. This reference sits inside the structural diffing & sync engines stage and specializes it for the moment the engine stops comparing and starts recording. It is written for the data engineers, migration specialists, Python pipeline builders, and platform operations teams for whom a discrepancy is not real until it is written down in a form a human, an automated re-sync, and an auditor can all trust.

A manifest is only useful if it is reproducible. The same two inputs, compared twice, must produce byte-identical manifests — otherwise the audit trail is fiction, a re-sync cannot be replayed, and two operators comparing notes see different discrepancies. That reproducibility is not free: it requires canonical ordering of records, a stable identity for each discrepancy that does not depend on iteration order, and a serialization that is deterministic down to key order and number formatting. Get those three right and the manifest becomes the authoritative source of truth the discrepancy routing and remediation stage can act on with confidence.

# Architectural Boundaries

This workload consumes the raw output of the comparison engines — the JSON and Parquet diffing results and the structural mismatch detection verdict — and produces a single serialized, hashed, optionally signed manifest. It owns record identity, ordering, and serialization; it does not decide what counts as a discrepancy (the tolerance policy does that upstream) nor what to do about one (routing does that downstream). Keeping generation free of both comparison and remediation logic is what makes the manifest a clean, replayable contract between the two.

# Prerequisites

• Every discrepancy has a stable identity. A discrepancy id must be a pure function of the record key, category, and schema version — never of iteration order or wall clock — so the same divergence gets the same id on every run.
• Canonical ordering rule chosen. A single, total ordering over discrepancies is declared (by key, then category) so two runs emit records in the same sequence.
• Deterministic serializer selected. The chosen format serializes with sorted keys and stable number formatting; any nondeterminism here breaks reproducibility.
• Schema and tolerance versions captured. The manifest records the schema version and tolerance profile that produced it, so a verdict can be interpreted years later.
• Signing key available (if required). For regulated use, a signing key and a NIST-approved hash are provisioned so the manifest is tamper-evident.
• Dependency libraries pinned. The serializer and hashing libraries are version-pinned so byte output is stable across hosts and releases.

# Step-by-Step Implementation

# 1. Assign a stable identity and category to each discrepancy

Identity is derived from the record key, the discrepancy category, and the schema version — nothing order-dependent — so a manifest is reproducible regardless of how the diff engine iterated.

import hashlib
import json
import logging
from dataclasses import dataclass, asdict
from enum import Enum
from typing import List, Optional

logging.basicConfig(
    format="%(asctime)s | %(levelname)s | %(name)s | %(message)s", level=logging.INFO
)
logger = logging.getLogger("recon.diffing.manifest")


class Category(str, Enum):
    ADDED = "added"
    DROPPED = "dropped"
    VALUE_DIVERGED = "value_diverged"


@dataclass(frozen=True)
class Discrepancy:
    key: str
    category: Category
    schema_version: str
    source_value: Optional[str] = None
    target_value: Optional[str] = None

    def discrepancy_id(self) -> str:
        material = f"{self.schema_version}|{self.category.value}|{self.key}"
        return hashlib.sha256(material.encode("utf-8")).hexdigest()[:24]

# 2. Sort into a canonical order and serialize deterministically

A total order over (category, key) plus sorted-key JSON makes the byte output identical across runs, which is the property the manifest digest depends on.

def serialize_manifest(discrepancies: List[Discrepancy], schema_version: str) -> str:
    """Deterministic manifest: canonical record order, sorted keys, stable formatting."""
    records = sorted(
        (
            {"id": d.discrepancy_id(), **asdict(d), "category": d.category.value}
            for d in discrepancies
        ),
        key=lambda r: (r["category"], r["key"]),      # total, reproducible order
    )
    envelope = {"schema_version": schema_version, "count": len(records), "records": records}
    return json.dumps(envelope, sort_keys=True, separators=(",", ":"), ensure_ascii=True)


def manifest_digest(serialized: str) -> str:
    """A reproducible fingerprint of the whole manifest for audit and tamper checks."""
    return hashlib.sha256(serialized.encode("utf-8")).hexdigest()

# 3. Emit a tamper-evident, versioned artifact

Hash the serialized manifest, optionally sign the digest, and write both to append-only storage so the manifest can be proven unaltered and replayed exactly.

def build_signed_manifest(discrepancies: List[Discrepancy], schema_version: str, sign) -> dict:
    """Return the serialized manifest, its digest, and a signature over the digest."""
    serialized = serialize_manifest(discrepancies, schema_version)
    digest = manifest_digest(serialized)
    logger.info("manifest built: %d record(s) digest=%s", len(discrepancies), digest[:12])
    return {"manifest": serialized, "digest": digest, "signature": sign(digest)}

# Serialization Format Trade-Off

Choose the format from the manifest’s readers and scale — a human triaging in a console has different needs than a petabyte-scale re-sync.

# Scaling and Performance

Manifest generation is O(discrepancies), and the sort dominates, so for very large deltas stream records through an external merge sort rather than materializing the whole set in memory, and shard by key range so each shard produces a sub-manifest that a deterministic merge combines. Emit JSON Lines for streamable, per-record-signable output at scale, reserving a single canonical JSON envelope for the smaller manifests humans actually read. Cache the schema-version and tolerance-profile headers so they are not recomputed per record, and compute the digest incrementally as records are written so a multi-gigabyte manifest is hashed in one pass rather than re-read. Because the digest is the reproducibility anchor, never let a non-deterministic field — a timestamp, a hostname, a run id — leak into the hashed body; carry those in an unhashed sidecar header instead.

# Failure Modes and Diagnostic Runbook

• Non-reproducible manifest. Cause: a non-deterministic field (timestamp, set iteration order) in the hashed body. Detection signal: two runs over identical inputs produce different digests. Remediation: move volatile fields to an unhashed header; enforce sorted-key serialization; the detail is in serializing deterministic discrepancy manifests.
• Identity collision. Cause: the identity function omits the schema version or category, so two distinct discrepancies share an id. Detection signal: record count drops after a dedup on id. Remediation: include schema version and category in the identity material.
• Unbounded memory on large deltas. Cause: the whole discrepancy set is sorted in memory. Detection signal: OOM on a large migration. Remediation: external merge sort with per-shard sub-manifests and a deterministic merge.
• Tampering undetected. Cause: the manifest is stored without a signed digest. Detection signal: an audit cannot prove a manifest is unaltered. Remediation: sign the digest with a NIST-approved hash and write to append-only storage.

# Deep Dives

• Serializing deterministic discrepancy manifests — the canonicalization rules, float and unicode normalization, and a byte-equality harness that guarantees reproducibility.

# Related

• Structural Diffing & Sync Engines — the parent stage whose comparison output this manifest records.
• Structural mismatch detection — supplies the structural verdicts that become manifest records.
• Discrepancy routing and remediation — the downstream consumer that acts on the manifest.
• Column-level checksum generation — the deterministic hashing discipline the manifest digest reuses.