Structural Mismatch Detection › Detecting Silent Schema Drift in Nested JSON

Detecting Silent Schema Drift in Nested JSON

This page answers a precise question: how do you detect that the shape of a nested JSON document has drifted — a field added three levels deep, a scalar quietly promoted to an object, an array that used to hold strings now holding numbers — before that drift silently corrupts a value comparison? It sits under the structural mismatch detection reference and specializes it for schemaless documents, where there is no footer to read and the schema is whatever the last writer happened to emit. This is the JSON analogue of the Parquet footer check, and it runs before any value-level JSON diffing is trusted.

Problem Framing

You reconcile documents flowing from a MongoDB collection into a JSON column in a warehouse. A service deploys a change that nests a field one level deeper and adds an optional block. Row counts match, top-level keys match, and the value comparison — which walks known paths — silently skips the new nesting, reporting parity while the migrated documents quietly lose the moved field. Because JSON carries no declared schema, drift like this is invisible until an analyst notices a column is empty. The defense is to derive a schema fingerprint from the set of type-annotated paths present across a sample and diff the fingerprints, so a moved or retyped field surfaces as a structural change rather than a value that happens to be absent.

Implementation

The extractor walks each document to a set of path: type entries, aggregates them across a sample into a fingerprint, and diffs two fingerprints into added, dropped, and retyped paths — with a tolerance for paths that are legitimately optional.

python
import logging
from dataclasses import dataclass, field
from typing import Any, Dict, Iterable, List, Set, Tuple

logging.basicConfig(
    format="%(asctime)s | %(levelname)s | %(name)s | %(message)s", level=logging.INFO
)
logger = logging.getLogger("recon.diffing.json_drift")


def type_paths(doc: Any, prefix: str = "") -> Set[Tuple[str, str]]:
    """Flatten a document into a set of (path, json_type) entries; arrays collapse by element type."""
    out: Set[Tuple[str, str]] = set()
    if isinstance(doc, dict):
        for k in doc:
            out |= type_paths(doc[k], f"{prefix}.{k}" if prefix else k)
    elif isinstance(doc, list):
        # Collapse array element types so [str,str] and [str] fingerprint the same.
        for item in doc:
            out |= type_paths(item, f"{prefix}[]")
        if not doc:
            out.add((f"{prefix}[]", "empty"))
    else:
        out.add((prefix, "null" if doc is None else type(doc).__name__))
    return out


@dataclass
class SchemaFingerprint:
    required: Set[Tuple[str, str]] = field(default_factory=set)   # in every sampled doc
    optional: Set[Tuple[str, str]] = field(default_factory=set)   # in some docs


def fingerprint(docs: Iterable[dict]) -> SchemaFingerprint:
    """Aggregate type-paths across a sample; paths present everywhere are required."""
    per_doc: List[Set[Tuple[str, str]]] = [type_paths(d) for d in docs]
    if not per_doc:
        return SchemaFingerprint()
    everywhere = set.intersection(*per_doc)
    anywhere = set.union(*per_doc)
    return SchemaFingerprint(required=everywhere, optional=anywhere - everywhere)

Key Implementation Notes

  • A path plus its type is the unit of drift. Tracking orders[].total: float rather than just orders catches a scalar-to-object promotion or an int-to-float retype that a key-only check misses entirely. The type annotation is what turns “the key exists” into “the shape is the same.”
  • Collapse array element types, or every list looks different. Fingerprinting each array position separately makes two documents with different list lengths always mismatch. Collapsing by element type compares the array’s schema, not its cardinality, which is what structural drift detection actually cares about.
  • Separate required from optional across a sample. A field present in some documents and absent in others is optional by nature; only a change to the required set — a path that used to be everywhere and now is not — is unambiguous drift. Diff the required sets strictly and treat optional churn as a warning, governed by the threshold tuning for tolerance profile.
  • Sample size bounds confidence, not correctness. A larger sample lowers the chance a rare optional field is misclassified as absent; record the sample size with the verdict so the confidence of a “no drift” claim is auditable.

Verification

Assert that a moved field, a retyped field, and a new required path are each caught, while benign optional variance is not.

python
base = [
    {"id": 1, "customer": {"name": "a"}, "total": 10.0},
    {"id": 2, "customer": {"name": "b"}, "total": 20.0},
]
drifted = [
    {"id": 3, "customer": {"name": "c", "tier": "gold"}, "total": 30},  # total int, +tier
    {"id": 4, "customer": {"name": "d", "tier": "silver"}, "total": 40},
]
fb, fd = fingerprint(base), fingerprint(drifted)
added = fd.required - fb.required
dropped = fb.required - fd.required
assert ("customer.tier", "str") in added          # new required nested field
assert ("total", "float") in dropped              # total retyped float -> int
assert ("total", "int") in added
logger.info("nested json drift detection verified")

Operational Considerations

Run drift detection on a bounded, representative sample per batch rather than every document, since the fingerprint converges quickly and a full scan is wasteful; size the sample to the tightest rare-field confidence the domain needs. Cache the last-known-good required set per source so drift is a cheap set difference rather than a re-derivation, and expose required_paths_added, required_paths_dropped, and retyped_paths so a shape change pages before it reaches the value comparison. Persist each fingerprint with its sample size and the source schema version, and when drift is confirmed, quarantine the batch and route it through the fallback chain implementation rather than letting a value diff run against a shape it does not understand.