Fallback Chain Implementation › Testing Diff Fallback Chains Under Partial Failure

Testing Diff Fallback Chains Under Partial Failure

This page answers a question that only matters at 3 a.m.: when a tier of your reconciliation fallback chain fails, does the chain actually degrade the way you designed, or does it crash, skip, or silently mislabel a partition as clean? It sits under the fallback chain implementation reference and treats the chain itself as the system under test. A fallback chain is only as good as its worst untested path, and the failure modes it exists to handle — a corrupt footer, a timed-out engine, a dead-letter write that itself fails — are exactly the ones that never happen in a demo and always happen in production.

Problem Framing

Your differ escalates through tiers: a fast columnar diff, then a hash comparison, then a query-based check, and finally quarantine with an alert. Each transition is triggered by a specific failure, and each tier has side effects — a metric, an audit record, a dead-letter write. The risk is that a transition is wrong in a way normal runs never exercise: tier 2 swallows an exception that should have escalated, quarantine fires but the audit record is never written, or two failures in a row hit an unhandled path. You cannot wait for real corruption to find these. You need deterministic fault injection that forces each failure and asserts both the verdict and the side effects at every tier.

Implementation

The harness wraps each tier with an injectable fault, drives the chain through every escalation path, and records which tier resolved the request plus the side effects that fired. Faults are deterministic — keyed by tier — so a failing test always reproduces.

python
import logging
from dataclasses import dataclass, field
from typing import Callable, Dict, List, Optional

logging.basicConfig(
    format="%(asctime)s | %(levelname)s | %(name)s | %(message)s", level=logging.INFO
)
logger = logging.getLogger("recon.diffing.fallback_test")


class TierError(Exception):
    """Raised by a tier to signal escalation to the next tier."""


@dataclass
class ChainResult:
    tier_reached: int
    verdict: str                       # "match", "diverged", "quarantined"
    side_effects: List[str] = field(default_factory=list)


@dataclass
class FallbackChain:
    tiers: List[Callable[[str], str]]  # each returns a verdict or raises TierError
    on_quarantine: Callable[[str], None]

    def run(self, partition: str, faults: Optional[Dict[int, Exception]] = None) -> ChainResult:
        """Escalate through tiers; inject a fault at a tier index to force its failure."""
        faults = faults or {}
        effects: List[str] = []
        for idx, tier in enumerate(self.tiers):
            try:
                if idx in faults:
                    raise faults[idx]
                verdict = tier(partition)
                effects.append(f"tier{idx}:resolved")
                return ChainResult(idx, verdict, effects)
            except TierError as exc:
                effects.append(f"tier{idx}:escalated")
                logger.warning("tier %d escalated for %s: %s", idx, partition, exc)
        # All tiers exhausted -> quarantine is mandatory, with its audit side effect.
        self.on_quarantine(partition)
        effects.append("quarantine:written")
        return ChainResult(len(self.tiers), "quarantined", effects)

Key Implementation Notes

  • Inject faults deterministically, keyed by tier. A flaky, randomized fault produces a test that fails once and passes on rerun, hiding the bug. Keying the injected exception to a tier index makes every escalation path reproducible and lets a failing case be pinned in a regression test.
  • Assert side effects, not just the verdict. The dangerous bugs are the ones where the verdict is right but a side effect is missing — quarantine returns “quarantined” but never wrote the audit record. Capture and assert the effect list so a silent omission fails the test.
  • Cover the double-failure paths. Most chains are tested for one failure at a time; the untested combinations — tier 1 and tier 2 both fail, or the quarantine write itself fails — are where crashes hide. Enumerate failure combinations, not just single faults.
  • Quarantine is mandatory, never optional. The exhausted-chain path must always quarantine and alert; a chain that can fall through to no verdict is worse than one that crashes, because it labels a partition as clean by omission. This is the invariant the fallback chain implementation reference defines.

Verification

Drive the chain through the happy path, each single-tier failure, and total failure, asserting the tier reached and the side effects.

python
tiers = [
    lambda p: (_ for _ in ()).throw(TierError("cold")),   # tier 0 always escalates here
    lambda p: "match",                                    # tier 1 resolves
]
quarantined: List[str] = []
chain = FallbackChain(tiers, on_quarantine=quarantined.append)

# Normal escalation: tier 0 escalates, tier 1 resolves.
r = chain.run("p1")
assert r.tier_reached == 1 and r.verdict == "match"
assert r.side_effects == ["tier0:escalated", "tier1:resolved"]

# Both tiers fail -> quarantine fires with its audit side effect.
r = chain.run("p2", faults={1: TierError("engine down")})
assert r.tier_reached == 2 and r.verdict == "quarantined"
assert "quarantine:written" in r.side_effects and quarantined == ["p2"]
logger.info("fallback chain partial-failure tests passed")

Operational Considerations

Run the fault-injection suite in CI so a change to any tier that breaks an escalation path fails the build, and periodically run it as a game-day drill against staging with real dependencies stubbed to fail, since a mock that always behaves cannot reveal a timeout-handling bug. Track fallback_tier_reached as a production histogram so a creeping rise in deep-tier resolutions signals a degrading upstream before it becomes an incident, and alert on any quarantine:written in steady state. Persist every quarantine with the fault that caused it and the tiers traversed, so a regulated review can prove why a partition took the degraded path — the same audit discipline the delta manifest generation stage applies to verdicts.