Cross-Engine Data Reconciliation Architecture › Reconciliation Strategy Decisions

Reconciliation Strategy Decisions

Reconciliation strategy decisions are the choices made before a single line of comparison logic is written: whether to reconcile in scheduled batches or a continuous stream, and which compute engine — PySpark, Dask, or Polars — should carry the workload. This reference sits inside the cross-engine data reconciliation architecture and exists because these two decisions constrain everything downstream, and reversing them after a pipeline is in production is expensive. It is written for the data engineers, migration specialists, Python pipeline builders, and platform operations teams who are asked “how should we build this?” and need an answer grounded in latency requirements, data volume, and operational cost rather than fashion.

The decisions are coupled but distinct. The batch-versus-streaming choice is about when reconciliation runs and how fresh its verdict must be; the engine choice is about how the comparison scales and what it costs to operate. Choosing wrong in either dimension produces a pipeline that technically works and quietly fails its real requirement — a nightly batch that cannot meet a cutover’s freshness need, or a distributed cluster spun up to reconcile a dataset that fits comfortably in memory on one node. This reference makes both choices deliberate, with a framework you can defend in a design review.

Decision path for reconciliation cadence and engine A first decision asks whether verdict freshness must be under a minute. If yes, the path leads to streaming reconciliation. If no, a second decision asks whether the dataset fits in memory on one node. If yes, the path leads to a single-node engine like Polars. If no, it leads to a distributed engine like PySpark or Dask for batch reconciliation. yes no yes no Freshness needed under a minute? Fits in memory on one node? Streaming continuous Single-node Polars batch Distributed PySpark / Dask batch
Two questions set the architecture: freshness under a minute routes to streaming; otherwise a memory-fit test chooses between a single-node engine and a distributed batch engine.

Architectural Boundaries

This workload is a design-time decision, not a runtime stage: it consumes the non-functional requirements — freshness, volume, cost ceiling, compliance posture — and produces a chosen cadence and engine that the rest of the architecture is built against. It does not implement comparison; it selects the frame in which comparison happens. The boundary matters because a strategy decision made implicitly, by whoever wrote the first prototype, becomes an accidental architecture that no one chose and everyone lives with.

Prerequisites

Step-by-Step Implementation

1. Score the cadence decision

Reduce the batch-versus-streaming choice to the freshness requirement and source capability, with volume as a tie-breaker. The framework is deterministic so two engineers reach the same answer from the same inputs.

python
import logging
from dataclasses import dataclass

logging.basicConfig(
    format="%(asctime)s | %(levelname)s | %(name)s | %(message)s", level=logging.INFO
)
logger = logging.getLogger("recon.arch.strategy")


@dataclass(frozen=True)
class Requirements:
    freshness_seconds: int
    rows_per_run: int
    node_memory_rows: int          # rows that fit comfortably in one node's memory
    has_change_stream: bool


def choose_cadence(req: Requirements) -> str:
    """Batch vs streaming from freshness and source capability."""
    if req.freshness_seconds <= 60 and req.has_change_stream:
        return "streaming"
    if req.freshness_seconds <= 60 and not req.has_change_stream:
        logger.warning("sub-minute freshness required but no change stream; use micro-batch")
        return "micro_batch"
    return "batch"

2. Score the engine decision

For batch workloads, choose the engine from the memory-fit test first — the single most predictive signal — then from ecosystem and cost.

python
def choose_engine(req: Requirements, cadence: str) -> str:
    """Pick a compute engine for a batch/micro-batch reconciliation workload."""
    if cadence == "streaming":
        return "stream_processor"          # engine choice folds into the streaming framework
    if req.rows_per_run <= req.node_memory_rows:
        return "polars"                    # single-node, lowest overhead when it fits
    if req.rows_per_run <= req.node_memory_rows * 50:
        return "dask"                      # scale out with a Python-native API
    return "pyspark"                       # largest scale, richest connector ecosystem

3. Record the decision with its inputs

A strategy decision is only defensible if the inputs that produced it are captured, so a later review can see why the architecture is what it is.

python
def record_decision(req: Requirements) -> dict:
    cadence = choose_cadence(req)
    engine = choose_engine(req, cadence)
    decision = {"cadence": cadence, "engine": engine, "inputs": req.__dict__}
    logger.info("strategy decision: %s on %s", cadence, engine)
    return decision

Strategy Trade-Off Matrix

The two decisions interact; this matrix scores the common combinations against what actually matters in operation.

Combination Freshness Cost Operational load Compliance / regulatory
Batch on Polars Hours Lowest Low; a scheduled job Strong when the reconcile window meets the retention rule.
Batch on PySpark/Dask Hours Moderate–high Moderate; a managed cluster Strong; mature audit and lineage tooling.
Micro-batch Minutes Moderate Moderate Strong; bounded, auditable intervals.
Streaming Seconds Highest steady-state High; a long-lived service Strongest for fixed-window regulated reconciliation.

Scaling and Performance

The decisions scale in opposite directions and should be revisited as the workload grows: a dataset that fits in memory today may cross the single-node boundary next quarter, flipping the engine choice from Polars to Dask, while a freshness requirement that tightens under a new SLA can flip cadence from batch to streaming. Treat both as parameters, not constants — re-run the framework on the current numbers each planning cycle rather than inheriting last year’s answer. When volume is the constraint, prefer scaling the batch engine before reaching for streaming, because distributed batch is operationally cheaper than a continuous service; reach for streaming only when freshness, not volume, is the binding requirement. The detailed engine benchmarks live in PySpark vs Dask vs Polars for reconciliation, and the cadence trade-off in depth in batch reconciliation vs streaming reconciliation.

Failure Modes and Diagnostic Runbook

  • Over-engineered streaming. Cause: streaming chosen for a workload whose real freshness need is hours. Detection signal: a continuous service idles most of the time at high steady-state cost. Remediation: move to scheduled batch; reserve streaming for genuine sub-minute needs.
  • Under-provisioned single node. Cause: Polars chosen for a dataset that outgrew memory. Detection signal: OOM or swap thrash on the reconcile host. Remediation: cross the boundary to Dask or PySpark; re-run the memory-fit test.
  • Distributed overhead on a small job. Cause: PySpark chosen for a dataset that fits in memory. Detection signal: cluster startup dominates runtime; per-run cost dwarfs the work. Remediation: collapse to a single-node engine.
  • Freshness cliff at cutover. Cause: a batch cadence that cannot meet a migration’s freshness window. Detection signal: the cutover waits on the next batch. Remediation: switch the cutover window to streaming or micro-batch reconciliation.

Deep Dives