Cross-Engine Data Reconciliation Architecture › Discrepancy Routing and Remediation

# Discrepancy Routing and Remediation

Discrepancy routing and remediation is the stage where reconciliation stops observing and starts acting: the workload that takes a delta manifest and decides, per discrepancy, whether to auto-correct it, quarantine it for later, or escalate it to a human. This reference sits inside the cross-engine data reconciliation architecture and is the last mile of the control plane — the point at which a detected divergence becomes a resolved one. It is written for the data engineers, migration specialists, Python pipeline builders, and platform operations teams who must ensure that finding a discrepancy actually leads to fixing it, without a well-meaning auto-remediation making things worse.

The stage exists because not every discrepancy is equal, and treating them uniformly fails in both directions. Auto-correcting everything risks a remediation loop that overwrites good data with bad; escalating everything buries a real incident under noise a human cannot triage. The discipline is severity-aware routing: classify each discrepancy by how confident and how consequential it is, send the safe and confident ones to idempotent auto-resync, the uncertain ones to a dead-letter queue for deferred handling, and the consequential ones to human review — every action recorded for audit.

# Architectural Boundaries

This workload consumes the serialized delta manifest and produces routed, acted-upon outcomes plus an audit record per discrepancy. It owns the routing decision and the remediation action; it does not decide what counts as a discrepancy (the comparison and tolerance stages do) nor how the manifest is produced. The boundary keeps remediation from silently re-defining parity: routing acts on the verdicts it is given, and any disagreement with those verdicts is a bug in the classifier’s policy, not a quiet correction.

# Prerequisites

• Severity policy defined. A rule maps each discrepancy category to a confidence and consequence score, so routing is deterministic rather than ad hoc.
• Remediation is idempotent. Every auto-resync is keyed on the discrepancy id so re-running it cannot double-apply, per the exactly-once discipline.
• Dead-letter store provisioned. An access-controlled queue holds deferred discrepancies with their full context for later handling.
• Escalation channel wired. A path to human review exists with enough context to triage without re-running the reconciliation.
• Audit sink is append-only. Every routing and remediation action writes an immutable record, since remediation modifies data and must be fully traceable.
• Loop guard configured. A discrepancy that reappears after remediation is detected and stopped rather than re-remediated indefinitely.

# Step-by-Step Implementation

# 1. Classify by confidence and consequence

Routing turns on two axes: how sure the pipeline is that the discrepancy is real and correctable, and how much damage a wrong correction would do. The classifier is a pure function of the discrepancy and policy, so it is testable and reproducible.

import logging
from dataclasses import dataclass
from enum import Enum

logging.basicConfig(
    format="%(asctime)s | %(levelname)s | %(name)s | %(message)s", level=logging.INFO
)
logger = logging.getLogger("recon.arch.routing")


class Route(Enum):
    AUTO_RESYNC = "auto_resync"
    DEAD_LETTER = "dead_letter"
    HUMAN_REVIEW = "human_review"


@dataclass(frozen=True)
class Discrepancy:
    id: str
    category: str            # e.g. "missing_in_target", "value_diverged"
    confidence: float        # 0..1, how sure the verdict is
    consequence: str         # "low", "high" (e.g. money/PII)


def classify(d: Discrepancy) -> Route:
    """Route by confidence and consequence; consequential cases always see a human."""
    if d.consequence == "high":
        return Route.HUMAN_REVIEW
    if d.confidence >= 0.95:
        return Route.AUTO_RESYNC
    return Route.DEAD_LETTER

# 2. Remediate idempotently

An auto-resync applies the source-of-truth value to the target keyed on the discrepancy id, so a retry or a replay is a no-op. It never runs for a high-consequence discrepancy.

@dataclass
class Remediator:
    apply_correction: callable      # (discrepancy_id, correction) -> None, idempotent
    _done: set = None

    def __post_init__(self):
        self._done = set()

    def resync(self, d: Discrepancy, correction: dict) -> bool:
        if d.id in self._done:
            logger.info("remediation already applied id=%s", d.id)
            return False
        self.apply_correction(d.id, correction)   # keyed on id -> idempotent at the sink
        self._done.add(d.id)
        logger.info("auto-resync applied id=%s", d.id)
        return True

# 3. Route, act, and audit

Every discrepancy flows through classification to its path, and every action — including a decision to defer or escalate — is recorded before control returns.

def route_and_record(d: Discrepancy, remediator: Remediator, audit: callable, get_correction) -> Route:
    route = classify(d)
    if route is Route.AUTO_RESYNC:
        remediator.resync(d, get_correction(d.id))
    audit({"id": d.id, "route": route.value, "category": d.category})
    return route

# Routing Policy Trade-Off

The aggressiveness of auto-remediation is a deliberate risk choice.

# Scaling and Performance

Classification is O(discrepancies) and stateless, so it parallelises trivially; the constraint is the remediation sink’s write capacity and the human-review channel’s throughput, neither of which scales like compute. Rate-limit auto-resync so a large manifest cannot storm the target, and batch dead-letter writes so a spike of uncertain discrepancies does not overwhelm the queue. Keep the loop guard’s seen-set bounded with a TTL keyed on the reconciliation interval so a re-appearing discrepancy is caught without the guard growing without limit. Shard routing by key range for very large manifests, and prioritise high-consequence discrepancies to the front of the review channel so the most important cases are triaged first regardless of manifest order.

# Failure Modes and Diagnostic Runbook

• Remediation loop. Cause: an auto-resync that does not actually fix the divergence, so the discrepancy reappears each run. Detection signal: the same discrepancy id recurs across consecutive runs. Remediation: the loop guard routes a re-appearing id to human review; fix the root cause before re-enabling auto-resync.
• Escalation overload. Cause: the consequence threshold is set too broad, sending routine cases to humans. Detection signal: review queue depth grows unbounded. Remediation: tighten the consequence policy; move confidently-safe categories back to auto-resync.
• Dead-letter black hole. Cause: deferred discrepancies are enqueued but never processed. Detection signal: dead-letter depth rises monotonically. Remediation: follow routing discrepancies to dead-letter queues for reprocessing SLAs and alerting.
• Unaudited correction. Cause: a remediation path that acts before writing its audit record. Detection signal: data changed with no matching audit entry. Remediation: make the audit write a precondition of the action, never a follow-up.

# Deep Dives

• Routing discrepancies to dead-letter queues — dead-letter schema, reprocessing SLAs, poison-message handling, and alerting on queue depth.

# Related

• Cross-Engine Data Reconciliation Architecture — the control plane this stage completes.
• Delta manifest generation — the manifest this stage consumes.
• Exactly-once reconciliation with idempotent sinks — the idempotency remediation depends on.
• PCI-DSS audit trail format for reconciliation — the tamper-evident record every remediation writes.