Implementing Salted SHA-256 Tokenization for Student IDs

Tokenizing a student identifier is the single mechanical operation that makes the FERPA-Compliant PII Handling boundary real: it turns a raw sis_user_id into a stable, opaque stu_… token that is safe to store, join, and log, while the mapping back to the real learner stays locked in a restricted vault. This guide implements that operation properly — with a keyed HMAC rather than a bare hash, a salt pulled from a secrets manager, versioned salts so rotation does not shatter longitudinal joins, and a separate reversible vault for the rare legitimate reversal. The token this produces is the join key the whole warehouse relies on, including the Cross-LMS Student ID Mapping that reconciles one learner across platforms.

The task is precise: given a raw student identifier, produce a deterministic token such that the same input always yields the same token (so it joins), a different salt yields a different token (so rotation works), and the token is computationally irreversible without the salt (so a leaked warehouse is not a leaked roster).

Prerequisites

Confirm each of these before running the procedure — the script assumes all of them.

Step-by-step implementation

1. Load the salt from a secrets manager, not from code. The salt is the secret that makes the token irreversible; if it lives in the repository or in analytical storage, the boundary is fictional. Read it at startup from the secrets tier and keep it in memory only. Here we read TOKEN_SALT_V1 from the environment as a local stand-in.

2. Use HMAC-SHA256, not bare sha256. A plain hashlib.sha256(raw_id) is reversible for a known ID format: an attacker hashes every candidate ID and matches. hmac.new(salt, raw_id, sha256) keys the hash with the secret salt, so without the salt the token space is unsearchable. This is the difference between “hashed” and “safe.”

3. Version every salt. Store the salt under a version label (v1, v2) and embed that version in the token (stu_v1_…). Rotating a salt then means adding v2 and tokenizing new data under it while historical tokens keep v1 — so a rotation never silently changes an existing learner’s token and breaks its joins.

4. Make tokenization deterministic for joins. Given the same salt version and raw id, the function must be pure and stable. Determinism is what lets two independently-processed batches — a Canvas pull and a Moodle pull — produce identical tokens for the same learner and join cleanly, exactly what the canonical dim_learner requires.

5. Keep the reversible vault separate and access-controlled. Some legitimate needs (a registrar records request, an at-risk outreach) require reversing a token. Persist the token→raw mapping only in the restricted vault, never beside the analytical tokens, and treat every reversal as an audited event via the FERPA audit-log schema.

6. Batch-tokenize a DataFrame column and drop the raw one. In practice you tokenize a whole column at ingestion. Map the tokenizer across the raw identifier column, write the token column, and drop the raw column before the frame leaves memory so PII never reaches analytical storage.

Complete runnable code block

python
import os
import hashlib
import hmac
from dataclasses import dataclass, field

import pandas as pd


@dataclass
class SaltRing:
    """Holds versioned salts. In production each value comes from a secrets manager."""
    salts: dict[str, bytes]              # {"v1": b"...", "v2": b"..."}
    current: str                         # version used for NEW tokenization

    def key(self, version: str) -> bytes:
        try:
            return self.salts[version]
        except KeyError:
            raise KeyError(f"no salt registered for version {version!r}")


@dataclass
class StudentTokenizer:
    ring: SaltRing
    prefix: str = "stu"

    def tokenize(self, raw_id: str, version: str | None = None) -> str:
        """Deterministic, irreversible token. Same input + version -> same token."""
        version = version or self.ring.current
        salt = self.ring.key(version)
        digest = hmac.new(salt, str(raw_id).encode("utf-8"), hashlib.sha256).hexdigest()
        return f"{self.prefix}_{version}_{digest[:24]}"

    def tokenize_column(self, df: pd.DataFrame, raw_col: str,
                        token_col: str = "learner_token") -> pd.DataFrame:
        """Tokenize a whole column and DROP the raw identifier before returning."""
        if raw_col not in df.columns:
            raise KeyError(f"column {raw_col!r} not in frame")
        out = df.copy()
        out[token_col] = out[raw_col].map(
            lambda v: self.tokenize(v) if pd.notna(v) else pd.NA
        )
        return out.drop(columns=[raw_col])


@dataclass
class ReversibleVault:
    """Restricted store mapping token -> raw id. NEVER co-located with analytics."""
    _map: dict[str, str] = field(default_factory=dict)

    def record(self, token: str, raw_id: str) -> None:
        # Idempotent: determinism guarantees a re-record is identical.
        self._map[token] = str(raw_id)

    def reverse(self, token: str) -> str | None:
        # Every call here should also write an audit row in production.
        return self._map.get(token)


def load_salt_ring() -> SaltRing:
    """Read salts from the environment as a stand-in for a real secrets manager."""
    v1 = os.environ.get("TOKEN_SALT_V1")
    if not v1:
        raise RuntimeError("TOKEN_SALT_V1 not set — load the salt from your secrets manager")
    salts = {"v1": v1.encode("utf-8")}
    if os.environ.get("TOKEN_SALT_V2"):
        salts["v2"] = os.environ["TOKEN_SALT_V2"].encode("utf-8")
    current = "v2" if "v2" in salts else "v1"
    return SaltRing(salts=salts, current=current)


if __name__ == "__main__":
    tokenizer = StudentTokenizer(ring=load_salt_ring())
    vault = ReversibleVault()

    roster = pd.DataFrame({"sis_user_id": ["SIS-90233", "SIS-90233", "SIS-41007"],
                           "score": [88.0, 88.0, 91.5]})

    # Populate the restricted vault BEFORE dropping the raw column.
    for raw in roster["sis_user_id"].unique():
        vault.record(tokenizer.tokenize(raw), raw)

    tokenized = tokenizer.tokenize_column(roster, raw_col="sis_user_id")
    print(tokenized)                                   # only learner_token + score remain
    sample = tokenizer.tokenize("SIS-90233")
    print("stable:", tokenizer.tokenize("SIS-90233") == sample)   # True — deterministic
    print("reverse:", vault.reverse(sample))           # 'SIS-90233' — vault only

Verification and output validation

Confirm the tokenizer behaves correctly before wiring it into ingestion:

  • Determinism (same input → same token). assert tokenizer.tokenize("SIS-90233") == tokenizer.tokenize("SIS-90233") — a stable token is what makes joins work.
  • Salt-sensitivity (different salt → different token). Build a second StudentTokenizer whose SaltRing has a different v1 salt and assert its token for "SIS-90233" differs. If they match, the salt is not actually feeding the HMAC.
  • No raw column survives. assert "sis_user_id" not in tokenized.columns — the raw identifier must not reach analytical storage.
  • Token shape. assert tokenized["learner_token"].dropna().str.match(r"stu_v\d+_[0-9a-f]{24}").all() — the version prefix must be present so rotation stays deterministic.
  • Reversal is vault-only. assert vault.reverse(sample) == "SIS-90233" and confirm the analytical frame has no column from which the raw id could be recovered.
  • Version isolation. Tokenize the same id under "v1" and "v2" and assert the two tokens differ — proof that a rotation produces a distinct, migratable token space.

Troubleshooting

  • RuntimeError: TOKEN_SALT_V1 not set. The salt is not loaded. Export it locally (export TOKEN_SALT_V1=…) or wire load_salt_ring() to your secrets manager. Never fall back to a hard-coded default — that silently makes every token reversible.
  • Tokens differ between two pipeline runs for the same student. The salt or its version changed between runs, or one run used bare sha256 and the other HMAC. Pin the salt_version and confirm both ingress points share one SaltRing; inconsistent tokens fragment the canonical learner, the failure covered in FERPA-Compliant PII Handling.
  • A joined analytical query returns zero matches across platforms. Canvas and Moodle tokenized the same learner under different salts, so tokens do not align. Use one salt version at every ingress; this is the join contract Cross-LMS Student ID Mapping relies on.
  • KeyError: no salt registered for version 'v2'. A token carries a version whose salt is not in the ring — common after restoring old data post-rotation. Keep every historical salt version in the ring; you retire raw salts, never versions still referenced by live tokens.
  • Raw IDs appearing in the vault-less analytical store. You tokenized but forgot to drop the raw column, or recorded to the vault after promoting the frame. Always drop the raw column in tokenize_column, and populate the vault only inside the restricted zone.
  • TypeError mapping over a column with mixed types. Some IDs arrive as integers, others as strings, so SIS-90233 and 90233 tokenize differently. Normalize the raw id to a canonical string form (strip, casefold, consistent prefix) before tokenizing.

Part of: FERPA-Compliant PII Handling