Cross-Engine Data Reconciliation Architecture › Security Boundaries for Reconciliation › Securing Reconciliation Pipelines in Multi-Cloud

# Securing Reconciliation Pipelines in Multi-Cloud Environments

This page answers one precise question: how do you prove that two engines hold the same logical data when the source and target live in different clouds, and a data-residency rule forbids the raw rows from being shipped to a single place to compare them? It extends the read-only, least-privilege boundary developed in the parent Security Boundaries for Reconciliation reference across an account and provider seam, where you can no longer assume one KMS, one IAM plane, or one network you control end to end. The prerequisite is that both sides already produce canonicalized row streams under a shared data equivalence modeling contract — without that, cross-cloud digests will diverge for reasons that have nothing to do with a real discrepancy. It is written for migration specialists, Python pipeline builders, and platform operations teams running a cutover whose two halves sit on different providers.

# Problem Framing: A Ledger Split Across AWS and GCP

Concretely: you are reconciling a 4-billion-row financial ledger during a phased migration. The source of truth is Aurora PostgreSQL in AWS us-east-1; the target is BigQuery in GCP europe-west4, because the EU business unit that will own the data going forward is being onboarded first. Two constraints collide. First, GDPR data-residency terms mean EU-subject rows may not be copied into a US region, and the security review forbids exporting the US ledger rows into the EU wholesale just to diff them. Second, cross-cloud egress is metered — pulling 4 billion raw rows out of both providers into a neutral comparison cluster would cost more in transfer than the migration itself and would create exactly the untrusted-transit exposure the boundary exists to prevent.

The resolution is the defining move of multi-cloud reconciliation: move the computation to the data, not the data to the computation. Each side hashes its own partitions inside its own cloud and region, under credentials brokered for that provider alone, and only the resulting digests — which carry no personal data — cross the seam to a comparison coordinator. The topology below shows the trust boundaries; note that no raw-row path ever leaves a cloud.

# Implementation

The build has two roles. A region-local digest worker runs inside each cloud, leases a credential scoped to that provider, streams its partition through a deterministic hasher, and signs the result with a KMS key that also lives in that region. A comparison coordinator — which may run anywhere, because it only ever sees digests — verifies both signatures and compares them. This is the same streaming, order-stable normalization discipline used for column-level checksum generation, lifted to operate independently on each side of a provider seam.

# Region-local digest worker

Each worker fetches rows in bounded batches so peak memory is one chunk regardless of partition size, coerces every batch to the pinned equivalence schema so schema drift is caught before it silently corrupts a digest, and never materializes the full partition. The credential is leased at runtime from that cloud’s broker and is never written to disk.

from __future__ import annotations

import hashlib
import hmac
import logging
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from typing import Any, Iterator, Mapping, Protocol, Sequence

logger = logging.getLogger("reconciliation.multicloud")

# Columns that carry no cross-engine meaning and must be excluded from the digest.
_ENGINE_METADATA = frozenset({"_id", "row_version", "_ingest_ts"})


class CredentialBroker(Protocol):
    """Provider-specific adapter: AWS STS AssumeRole, GCP Workload Identity, etc."""
    def lease(self, scope: str, ttl_minutes: int) -> "CloudCredential": ...


class RegionalSigner(Protocol):
    """Signs a digest with a key that physically resides in this worker's region."""
    def sign(self, message: bytes) -> bytes: ...
    def key_id(self) -> str: ...


@dataclass(frozen=True)
class CloudCredential:
    principal: str
    secret: str
    lease_expiry: datetime

    def assert_valid(self, skew_seconds: int = 30) -> None:
        remaining = (self.lease_expiry - datetime.now(timezone.utc)).total_seconds()
        if remaining <= skew_seconds:
            raise RuntimeError(
                f"Credential for {self.principal} expires in {remaining:.0f}s; refusing to start"
            )


@dataclass(frozen=True)
class PartitionDigest:
    """The only artifact allowed to cross the cloud seam — carries no raw data."""
    partition_id: str
    cloud: str
    region: str
    schema_version: str
    row_count: int
    digest: str
    key_id: str
    signature: str  # hex HMAC/KMS signature over the canonical body

    def signing_message(self) -> bytes:
        return "|".join(
            (self.partition_id, self.cloud, self.region, self.schema_version,
             str(self.row_count), self.digest)
        ).encode()


def _normalize(value: Any) -> bytes:
    """Deterministic, engine-independent normalization so equal rows hash equally
    on both providers. Pins float format and coerces all timestamps to UTC."""
    if value is None:
        return b"__NULL__"
    if isinstance(value, datetime):
        return value.astimezone(timezone.utc).isoformat().encode()
    if isinstance(value, float):
        return f"{value:.6f}".encode()
    return str(value).encode()


def compute_partition_digest(
    *,
    partition_id: str,
    cloud: str,
    region: str,
    schema_version: str,
    rows: Iterator[Mapping[str, Any]],
    signer: RegionalSigner,
    chunk_bytes: int = 8192,
) -> PartitionDigest:
    """Stream a partition through a bounded BLAKE2b hasher inside one cloud/region
    and return a region-signed digest. Raw rows never leave this function."""
    hasher = hashlib.blake2b(digest_size=32)
    buffer = bytearray()
    count = 0
    for row in rows:
        payload = b"|".join(
            _normalize(row[k]) for k in sorted(row.keys()) if k not in _ENGINE_METADATA
        )
        buffer.extend(payload)
        count += 1
        if len(buffer) >= chunk_bytes:
            hasher.update(buffer)
            buffer.clear()
    if buffer:
        hasher.update(buffer)

    digest = hasher.hexdigest()
    unsigned = PartitionDigest(
        partition_id=partition_id, cloud=cloud, region=region,
        schema_version=schema_version, row_count=count, digest=digest,
        key_id=signer.key_id(), signature="",
    )
    signature = signer.sign(unsigned.signing_message()).hex()
    logger.info(
        "digest partition=%s cloud=%s region=%s rows=%d key=%s",
        partition_id, cloud, region, count, signer.key_id(),
    )
    return PartitionDigest(**{**unsigned.__dict__, "signature": signature})

# Cross-cloud comparison coordinator

The coordinator is the only component that spans the seam, and it is deliberately powerless: it holds no engine credentials, issues no reads, and sees no rows. It verifies each side’s signature against that region’s public key material, checks that both digests describe the same partition under the same schema version, and emits a verdict against the divergence budget defined in threshold tuning for tolerance.

from typing import Literal

Verdict = Literal["MATCHED", "DIVERGED", "INDETERMINATE"]


class SignatureVerifier(Protocol):
    def verify(self, key_id: str, message: bytes, signature: bytes) -> bool: ...


class ResidencyViolation(RuntimeError):
    """Raised when a digest arrives from a region the job is not authorized to compare."""


def compare_across_clouds(
    source: PartitionDigest,
    target: PartitionDigest,
    verifier: SignatureVerifier,
    authorized_regions: Sequence[str],
) -> Verdict:
    for side in (source, target):
        if side.region not in authorized_regions:
            raise ResidencyViolation(
                f"partition {side.partition_id} signed in unauthorized region {side.region}"
            )
        if not verifier.verify(side.key_id, side.signing_message(),
                               bytes.fromhex(side.signature)):
            logger.error("signature verification FAILED partition=%s cloud=%s",
                         side.partition_id, side.cloud)
            return "INDETERMINATE"

    if source.schema_version != target.schema_version:
        logger.warning("schema_version mismatch %s != %s",
                       source.schema_version, target.schema_version)
        return "INDETERMINATE"

    verdict: Verdict = "MATCHED" if source.digest == target.digest else "DIVERGED"
    logger.info("compare partition=%s verdict=%s src=%s tgt=%s",
                source.partition_id, verdict, source.cloud, target.cloud)
    return verdict

# Key Implementation Notes

• BLAKE2b over MD5, and a signature over the digest. BLAKE2b is fast and collision-resistant, which matters when each side is hashing billions of rows locally. The signature is the multi-cloud addition: because the digest travels across a seam you do not fully control, the coordinator must be able to prove which region produced it before trusting it — a bare digest could be replayed or forged. Align the signing key lifecycle with NIST SP 800-57 guidance, and give each region its own key so a compromise on one provider cannot forge the other side’s evidence.
• Region-pinned keys are a residency control, not just a security one. Signing in-region means the key material and the hashing both stay inside the jurisdiction that owns the data. The authorized_regions guard in the coordinator turns “rows must not cross” from a policy into an enforced invariant: a digest signed in a region the job was not scoped for raises ResidencyViolation instead of being silently compared.
• The equivalence contract must be version-pinned on both sides. schema_version is compared explicitly and a mismatch yields INDETERMINATE, never a false DIVERGED. When a source column type is promoted (say INT to BIGINT), the correct response is a coordinated re-pin sourced from schema validation pre-checks, not a digest that quietly disagrees. Cross-provider type quirks are cataloged in the cross-platform schema mapping reference.
• Clock skew is a real cross-cloud failure vector. Time-windowed parity checks break when AWS and GCP nodes disagree on now. Anchor every timestamp to UTC inside _normalize and gate the pipeline on measured drift (see verification below) rather than assuming the two providers’ clocks agree.
• The coordinator is intentionally credential-free. Keeping engine reads inside each cloud means the one component that spans providers can never be turned into an actor. This preserves the read-only, evidence-only posture of the boundary even though the seam itself is not a trusted network.

# Verification

Assert the two properties that make cross-cloud comparison trustworthy: that logically identical partitions hash identically regardless of key order or provider, and that an unauthorized region is rejected rather than compared.

class _FakeSigner:
    def __init__(self, key: bytes, kid: str): self._k, self._kid = key, kid
    def sign(self, message: bytes) -> bytes:
        return hmac.new(self._k, message, hashlib.sha256).digest()
    def key_id(self) -> str: return self._kid


class _FakeVerifier:
    def __init__(self, keys: dict[str, bytes]): self._keys = keys
    def verify(self, key_id, message, signature) -> bool:
        expected = hmac.new(self._keys[key_id], message, hashlib.sha256).digest()
        return hmac.compare_digest(expected, signature)


aws = _FakeSigner(b"aws-region-key", "kms/us-east-1")
gcp = _FakeSigner(b"gcp-region-key", "kms/europe-west4")

left = [{"id": 1, "amount": 9.5}, {"id": 2, "amount": 3.0}]
right = [{"amount": 9.5, "id": 1}, {"amount": 3.0, "id": 2}]  # same rows, shuffled keys

src = compute_partition_digest(partition_id="p-1", cloud="aws", region="us-east-1",
                               schema_version="v3", rows=iter(left), signer=aws)
tgt = compute_partition_digest(partition_id="p-1", cloud="gcp", region="europe-west4",
                               schema_version="v3", rows=iter(right), signer=gcp)

verifier = _FakeVerifier({"kms/us-east-1": b"aws-region-key",
                          "kms/europe-west4": b"gcp-region-key"})
assert compare_across_clouds(src, tgt, verifier,
                             ["us-east-1", "europe-west4"]) == "MATCHED"

# A digest from an unauthorized region must be refused, not compared.
try:
    compare_across_clouds(src, tgt, verifier, ["us-east-1"])  # europe-west4 not allowed
    raise AssertionError("expected ResidencyViolation")
except ResidencyViolation:
    pass

Before a run, confirm the clocks the digests depend on are actually synchronized. On every worker node:

# Acceptable drift for time-windowed parity is < 50 ms; abort the run above it.
chronyc tracking | awk '/System time/ {print "drift_seconds:", $4}'

# Operational Considerations

The dominant cost in multi-cloud reconciliation is cross-cloud egress, and the region-local design is what keeps it flat: only fixed-size digests (tens of bytes per partition) traverse the seam, so transfer cost scales with partition count, not row count. Size partitions so a single leased session and one KMS signature amortize across tens of thousands of rows — the same batch-sizing discipline as async batching for large datasets, with a residency-driven floor so per-partition signing does not dominate. Because hashing is CPU-bound and holds the GIL, fan out across partitions with multiprocessing or separate worker pods per region rather than threads.

Where the comparison should physically run is itself a decision with a compliance dimension, so weigh it explicitly rather than defaulting to a neutral cluster:

Expose these signals to catch divergence before it cascades: per-partition verdict distribution (a rising INDETERMINATE rate usually means signature or schema-version trouble, not real drift), lease-remaining seconds against the skew threshold, measured clock drift per region, and digest-signing latency. Keep the audit trail append-only and payload-free — partition id, region, schema version, digests, verdict, and the signing key id — so it doubles as residency evidence without ever holding a row. When a partition comes back DIVERGED, route it to structural mismatch detection for a field-level diff, and let the fallback chain implementation govern how the pipeline degrades — buffer locally on a network partition, drop to hourly batch comparison if one cloud throttles — while continuing to emit auditable evidence.

# Related

• Security Boundaries for Reconciliation — the single-plane boundary this page extends across a provider seam: credential leasing, masking, and signed manifests.
• How to Validate SQL vs NoSQL Data Parity — the heterogeneous-engine parity runbook these digest workers feed.
• Generating MD5 vs SHA-256 Checksums for Data Rows — choosing the row-level hash the region-local workers compute.
• Threshold Tuning for Tolerance — the divergence budget the coordinator’s verdict is measured against.
• Fallback Chain Implementation — graceful degradation when a cloud throttles or the seam partitions.

Up one level: Security Boundaries for Reconciliation.

# Frequently Asked Questions