Security Boundaries for Reconciliation › GDPR Pseudonymisation in Reconciliation Pipelines

GDPR Pseudonymisation in Reconciliation Pipelines

This page answers a specific compliance question: how do you reconcile datasets containing personal data across two engines without ever moving the raw identifiers, while still being able to prove the two sides match? It sits under the security boundaries for reconciliation reference and applies its masking mandate to the GDPR-specific requirement of pseudonymisation — a reversible-by-a-held-key transform that lets a pipeline join and compare records by a stable pseudonym instead of a name, email, or national ID.

Problem Framing

You reconcile a customer table migrating from PostgreSQL to a document store, and the join key is an email address — personal data under the GDPR. Reconciling on the raw email would spray identifiers through logs, dead-letter queues, and comparison buffers, each a new place the data must be secured and later erased. Pseudonymisation solves this: replace each identifier with a keyed HMAC before it enters the pipeline, so the reconciler joins and hashes on a pseudonym that is stable (the same email always maps to the same pseudonym, on both engines) yet useless to anyone without the key. The key lives in a separate boundary the reconciler never reads, satisfying the GDPR’s requirement that pseudonymised data and the re-identification key be held apart.

Implementation

The transform is a keyed HMAC — deterministic so parity is preserved, keyed so it cannot be reversed by rainbow table, and salted per field so the same value in two columns does not collide. The key is fetched from a separate secret boundary, never logged.

python
import hmac
import hashlib
import logging
from dataclasses import dataclass
from typing import Callable, Dict

logging.basicConfig(
    format="%(asctime)s | %(levelname)s | %(name)s | %(message)s", level=logging.INFO
)
logger = logging.getLogger("recon.security.gdpr")


@dataclass(frozen=True)
class PseudonymConfig:
    field: str
    key_id: str                     # names a key in the separate secret store


def pseudonymise(value: str, cfg: PseudonymConfig, get_key: Callable[[str], bytes]) -> str:
    """Deterministic, keyed HMAC pseudonym: stable across engines, not reversible without the key."""
    if value is None:
        return ""                   # null maps to a fixed empty pseudonym, never hashed
    # Per-field domain separation so the same identifier in two fields differs.
    material = f"{cfg.field}:{value}".encode("utf-8")
    key = get_key(cfg.key_id)       # fetched from the secret boundary; never logged
    return hmac.new(key, material, hashlib.sha256).hexdigest()


def pseudonymise_row(row: Dict[str, str], configs, get_key) -> Dict[str, str]:
    """Replace configured PII fields with pseudonyms; pass non-PII fields through."""
    out = dict(row)
    for cfg in configs:
        if cfg.field in out:
            out[cfg.field] = pseudonymise(out[cfg.field], cfg, get_key)
    return out

Pseudonymisation Technique Trade-Off

The technique is chosen from whether re-identification must ever be possible and who holds the key.

Technique Parity preserved Reversible Key custody Compliance / regulatory
Keyed HMAC (this page) Yes; deterministic No, one-way Held apart from data Strong: GDPR pseudonymisation with separated re-identification key.
Format-preserving encryption Yes Yes, with key Held apart Strong; reversible when re-identification is a stated need.
Tokenization (vault) Yes Yes, via vault Central vault Strong; centralises re-identification and access logging.
Plain SHA-256 (unkeyed) Yes No None Weak: low-entropy identifiers fall to a dictionary attack; avoid for PII.

Key Implementation Notes

  • Keyed, never plain. An unkeyed hash of an email or a national ID is reversible by dictionary attack because the input space is small. A keyed HMAC with the key held separately is what makes the pseudonym defensible under the GDPR; use a NIST-approved hash under the HMAC.
  • Determinism is what preserves parity. The same identifier must produce the same pseudonym on both engines, or the reconciler’s join breaks and every row reads as a discrepancy. HMAC is deterministic given the key, which is exactly why it can replace the raw key without losing comparability.
  • Separate the key boundary from the pipeline. The reconciler holds pseudonyms; a distinct secret store holds the key. Re-identification requires crossing a boundary the reconciler cannot cross alone — the separation the regulation demands and the security boundaries for reconciliation reference enforces.
  • Erasure propagates by pseudonym. A right-to-erasure request deletes the mapping and every pseudonymised record; because the pseudonym is deterministic, the same request key locates the record on both engines and in every dead-letter and audit store.

Verification

Assert that pseudonyms are stable across “engines” (same key), differ across fields, and are not the raw value.

python
keys = {"k-customer-2026": b"a-32-byte-secret-key-material-xx"}
get_key = keys.__getitem__
cfg = PseudonymConfig(field="email", key_id="k-customer-2026")

p1 = pseudonymise("alice@example.com", cfg, get_key)
p2 = pseudonymise("alice@example.com", cfg, get_key)   # a second engine, same key
assert p1 == p2 and p1 != "alice@example.com"          # stable + not raw

other = PseudonymConfig(field="backup_email", key_id="k-customer-2026")
assert pseudonymise("alice@example.com", other, get_key) != p1   # field separation
logger.info("gdpr pseudonymisation parity verified")

Operational Considerations

Fetch keys from a managed secret store with its own access log so every re-identification is itself auditable, and rotate keys on a schedule — but note that rotation changes every pseudonym, so run a dual-key window during reconciliation across a rotation boundary or pin the reconcile to a key version. Keep raw identifiers out of every downstream store by pseudonymising at extraction, before the data extraction hashing workflows stage buffers anything, so logs and dead-letter queues never hold PII in the first place. Record the key id (not the key) and the pseudonymisation config version with each reconciliation run so a regulator can verify which transform produced a given pseudonym, and treat the pseudonym-to-identifier mapping as the single most sensitive asset in the pipeline, with erasure that reaches it and every derived record.