Column-Level Checksum Generation › Choosing Hash Algorithms for PII Columns

# Choosing Hash Algorithms for PII Columns

This page answers a question where the wrong default is quietly dangerous: which hash algorithm should you use for a column containing personal data, when the same pipeline uses a fast non-cryptographic hash everywhere else? It sits under the column-level checksum generation reference and draws the line between hashing for parity — where speed wins — and hashing for PII, where a fast unkeyed hash is a data-exposure incident waiting to happen. The algorithm choice is not a performance decision for these columns; it is a compliance one.

# Problem Framing

Your reconciler hashes every column to compare source and target cheaply, and xxHash is the right tool for that: fast, well-distributed, perfect for detecting a value change. But when the column is an email, a national ID, or a phone number, that same hash becomes a liability. These identifiers are low-entropy — the space of valid emails or phone numbers is small enough to enumerate — so an unkeyed hash of them is reversible by dictionary attack. Anyone who obtains the hashes can recover the identifiers. For PII columns the requirement flips: the hash must be a keyed HMAC with a secret held apart from the data, so the fingerprint is stable enough to reconcile but useless to reverse. Using one algorithm for both purposes is the mistake.

# Implementation

The pipeline selects the algorithm from a per-column classification: a fast non-cryptographic hash for ordinary parity columns, and a keyed HMAC with a NIST-approved hash for PII columns, salted per field to prevent cross-column correlation.

import hmac
import hashlib
import logging
from dataclasses import dataclass
from enum import Enum
from typing import Callable

logging.basicConfig(
    format="%(asctime)s | %(levelname)s | %(name)s | %(message)s", level=logging.INFO
)
logger = logging.getLogger("recon.hashing.pii")


class Sensitivity(Enum):
    PARITY = "parity"     # ordinary column: speed wins
    PII = "pii"           # personal data: keyed, non-reversible


@dataclass(frozen=True)
class ColumnHashPolicy:
    field: str
    sensitivity: Sensitivity
    key_id: str = ""


def hash_value(value: str, policy: ColumnHashPolicy, get_key: Callable[[str], bytes]) -> str:
    """Fast hash for parity columns; keyed HMAC-SHA-256 for PII columns."""
    if value is None:
        return ""
    material = f"{policy.field}:{value}".encode("utf-8")   # per-field domain separation
    if policy.sensitivity is Sensitivity.PII:
        if not policy.key_id:
            raise ValueError(f"PII column {policy.field} requires a keyed hash")
        return hmac.new(get_key(policy.key_id), material, hashlib.sha256).hexdigest()
    # Parity column: a fast cryptographic digest is fine; reversibility is not a concern.
    return hashlib.blake2b(material, digest_size=16).hexdigest()

# Algorithm Decision Table

The right algorithm follows from what the column contains and what an attacker could do with the hash.

# Key Implementation Notes

• Low entropy is what makes an unkeyed PII hash reversible. An email or phone number comes from a small, enumerable space, so an attacker precomputes every hash and inverts it instantly. A secret key defeats this because the attacker cannot compute the HMAC without it — the same reasoning behind GDPR pseudonymisation.
• Salt per field to prevent correlation. Domain-separating the HMAC by field name stops the same identifier in two columns from producing the same hash, which would otherwise leak that the two columns hold the same value.
• Keep speed for the columns that do not need protection. Most columns are not PII, and forcing HMAC on all of them wastes the throughput that made column-level hashing cheap. Classify per column and reserve the keyed hash for the identifiers that actually need it.
• Key custody is the real control. The algorithm is only as strong as the key’s separation from the data; hold it in the boundary the security boundaries for reconciliation reference defines, and rotate it with a versioning plan since rotation changes every PII hash.

# Verification

Assert that a PII column’s hash is keyed (unguessable without the key), stable, and field-separated, and that a parity column stays fast.

keys = {"k-pii-2026": b"a-32-byte-secret-key-material-zz"}
get_key = keys.__getitem__
pii = ColumnHashPolicy("email", Sensitivity.PII, "k-pii-2026")
parity = ColumnHashPolicy("status", Sensitivity.PARITY)

h1 = hash_value("alice@example.com", pii, get_key)
h2 = hash_value("alice@example.com", pii, get_key)
assert h1 == h2 and h1 != hashlib.sha256(b"email:alice@example.com").hexdigest()  # keyed, not plain
# A PII column without a key must fail loudly, never silently fall back to a weak hash.
try:
    hash_value("x", ColumnHashPolicy("ssn", Sensitivity.PII), get_key)
    raise AssertionError("missing key should raise")
except ValueError:
    pass
assert hash_value("active", parity, get_key)      # parity column still hashes fast
logger.info("pii hash algorithm selection verified")

# Operational Considerations

Drive the sensitivity classification from a data catalog or a reviewed column policy rather than inline flags, so a compliance officer can audit exactly which columns are treated as PII, and fail the pipeline if a column marked PII lacks a key rather than letting it fall back to a weak hash. Fetch keys from a managed secret store with access logging, pin the key version across a reconciliation so a rotation does not break parity mid-run, and record the key id and algorithm per column with each verdict for reproducibility. Keep the fast path for parity columns so the throughput win of column-level hashing survives, and route any re-identification need through a reversible transform under key control rather than weakening the reconciliation hash.

# Related

• Column-level checksum generation — the parent reference whose hashing this specialises for PII.
• Generating MD5 vs SHA-256 checksums for data rows — the row-level algorithm choice this complements.
• GDPR pseudonymisation in reconciliation pipelines — the keyed-hash transform for identifiers.
• Security boundaries for reconciliation — the key-custody boundary this depends on.