How to Mask PII on Ingest Into an S3 Data Lake

The question almost never starts in engineering. It arrives from risk or legal and sounds like this: which columns in the lake hold customer identifiers, who can read them, and what happens when one of those customers asks to be deleted?

If answering that means opening a Glue job and reading Spark code, you have already lost the argument. The fix is structural: you mask PII in an S3 data lake at the boundary where data lands, before anything downstream reads it, deciding per column whether the masked value has to be reversible, joinable, or simply gone.

rule Decide what "masked" has to mean before writing any code

Masking projects go wrong at the vocabulary stage, because "masked", "anonymised" and "pseudonymised" get used interchangeably in one meeting and they are not the same thing.

The regulation is specific. Article 4(5) of the GDPR defines pseudonymisation as "the processing of personal data in such a manner that the personal data can no longer be attributed to a specific data subject without the use of additional information, provided that such additional information is kept separately and is subject to technical and organisational measures to ensure that the personal data are not attributed to an identified or identifiable natural person".

The second half of that sentence assumes the additional information still exists somewhere. Hashing an email with a secret pepper is pseudonymisation, not anonymisation, because your pipeline holds the pepper and the source system still holds the email. Pseudonymised columns stay in scope, so they still need access control, encryption and an erasure plan. Only genuinely irreversible columns fall out of the problem, which makes the cheapest column the one you never ingest. Before reaching for any technique below, walk the source schema and delete the fields nobody has a stated use for.

alt_route Where to mask PII in an S3 data lake

Mask between the landing bucket and the raw zone, not later.

The reason is copy arithmetic. Every zone, every extract and every ad-hoc CREATE TABLE AS SELECT multiplies the places an identifier lives. If unmasked values reach the raw zone, your erasure and audit surfaces grow with each new consumer and never shrink again. Masking at the boundary holds the blast radius at one bucket.

Three properties make the landing bucket safe enough to hold real identifiers for a few hours:

1. A lifecycle rule expiring objects within a day or two, so raw PII is never the long-lived copy.
2. A bucket policy granting s3:GetObject to the masking job's role and nothing else, no human principals.
3. CloudTrail data events on that prefix. Management events are on by default; object-level reads are not, and "who read the raw file" is exactly what an auditor asks.

On a tier-1 bank's Sybase to Redshift warehouse migration I worked on, extracts were staged through S3 before the bulk load into Redshift. That staging hop is precisely where a masking step belongs: it already exists, it is already a single choke point, and a transform there costs far less than a retrofit after fifty downstream tables have been built on unmasked columns.

table_chart Redaction, hashing, tokenisation or encryption?

Four techniques, four different answers to "can anyone get the original value back, and who". Pick per column, not per pipeline.

Two traps. A vault-based tokeniser is reversible and collision-free, but it puts a synchronous lookup inside a Spark job processing millions of rows: a throughput cliff and a new single point of failure. And deterministic encryption preserves joins while staying reversible, yet it leaks frequency. If 4% of your ciphertexts are identical, an observer learns 4% of your customers share a value, which on gender or product tier is close to no protection.

key Deterministic hashing that keeps your joins working

This is the technique most teams actually need, and the one most often implemented badly.

The requirement is that the same input produces the same output every time, in every job, across years, so orders.customer_key still joins to web_events.customer_key once the identifiers are gone. A plain SHA-256 satisfies that, and fails badly on any identifier with a guessable domain. There are only so many UK mobile numbers, and someone with a laptop can hash all of them and build a reverse lookup in an afternoon.

The fix is a secret: HMAC-SHA-256 with a pepper in Secrets Manager, readable only by the masking job's execution role and never present in the analytics account.

import hashlib
import hmac
from pyspark.sql import functions as F
from pyspark.sql.types import StringType
 
# Fetched once per job run. Never logged, never written to S3, and the
# analytics account has no IAM path to the secret that holds it.
PEPPER: bytes = get_secret("datalake/pii-pepper-v1")
 
def pseudonymise(value):
    if value is None:
        return None
    # Normalise FIRST. "Alice@Example.com " and "alice@example.com"
    # must produce the same key or your joins will silently under-match.
    normalised = value.strip().lower().encode("utf-8")
    return hmac.new(PEPPER, normalised, hashlib.sha256).hexdigest()
 
pseudonymise_udf = F.udf(pseudonymise, StringType())
 
masked = (
    raw
    .withColumn("customer_key", pseudonymise_udf(F.col("email")))
    .drop("email", "phone", "national_id")
)

Four rules keep this working in production:

1. Normalise before you hash. Case, whitespace and formatting differences produce different digests, and the failure mode is quiet: joins return fewer rows, nothing errors, a report is wrong for a year.
2. Version the pepper in the column or table name. Rotating it changes every key and breaks every historical join. Sometimes the right call, but a dated migration rather than a surprise.
3. Never hash a low-cardinality field. A two-value column gives you two digests and zero protection.
4. Use a separate pepper per environment, so production keys cannot be reproduced from a laptop holding the test secret.

If the Python UDF becomes the bottleneck, Spark's native sha2 avoids the serialisation cost, at the price of a prepended salt rather than a proper HMAC:

masked = raw.withColumn(
    "customer_key",
    F.sha2(F.concat(F.lit(PEPPER_HEX), F.lower(F.trim(F.col("email")))), 256),
)

Prefix-salted SHA-256 is weaker than HMAC in theory, and usually acceptable against this threat model. Put the trade-off in the design doc rather than leaving it implicit.

Glue offers hashing as a built-in action: its Detect PII transform can "pass the detected PII value to a SHA-256 cryptographic hash function and replace the value with the function's output". That built-in is unsalted, so treat it as fine for high-entropy values and inadequate for emails and phone numbers.

search Detecting the PII you did not know you had

Hand-written column lists rot. A new source adds a notes field, somebody pastes a national insurance number into it, and your masking config has no idea.

Glue's sensitive-data detection runs outside Glue Studio too, through two APIs: detect() scans every row and can mask in place, while classifyColumns() samples and returns a map of column to detected entity types. The sampling defaults are where teams pick up a false sense of coverage:

classifyColumns(frame, entityTypesToDetect,
                sampleFraction = 0.1,      # scans 10% of rows by default
                thresholdFraction = 0.1,   # column flagged only if 10%+ of
                                           # sampled rows contain the entity
                detectionSensitivity = "LOW")

Both default to 10%, so a column where 4% of rows carry a leaked identifier is never flagged. Reasonable for a discovery tool. Unreasonable for a compliance control.

There is a defaults mismatch too. The Studio documentation says High sensitivity is the default and that "all AWS Glue jobs created after November 2023 are automatically opted-in to this setting", while the programmatic signature defaults detectionSensitivity to "LOW". Console-authored and code-authored jobs can behave differently on identical data, so set it explicitly.

Use detection as an audit rather than as the control. Run classifyColumns() at sampleFraction = 1.0 on a schedule over the raw zone and alert on any column your masking list does not already know about.

lock KMS envelope encryption for fields you must read back

Some fields have to survive in recoverable form. A bank cannot hash the account number it prints on a statement.

For those, use envelope encryption rather than calling KMS per row. You ask KMS for a data key and it "returns a plaintext data key for immediate use (optional) and an encrypted copy of the data key that you can safely store with the data". Encrypt the column values locally, drop the plaintext key from memory, write the wrapped key alongside the partition.

The line that matters operationally sits in the same document: "AWS KMS does not store, manage, or track your data keys". Lose the wrapped key and you lose the data, so back it up and test the restore. Keep the shape in mind, because it pays off below: one data key per subject, or per cohort.

shield_person Column- and row-level access with Lake Formation

Masking decides what the bytes on disk are. Lake Formation decides who sees which of them at query time. You need both, and confusing the two is the most common architectural mistake here.

It works through data filters on a Data Catalog table, giving column-level, row-level and cell-level security. A filter pairs a column include/exclude list with a WHERE-style row expression:

{
  "Name": "analysts-no-direct-identifiers",
  "DatabaseName": "lake_curated",
  "TableName": "customers",
  "TableCatalogId": "111122223333",
  "RowFilter": { "FilterExpression": "region = 'UK'" },
  "ColumnWildcard": {
    "ExcludedColumnNames": ["postcode", "date_of_birth"]
  }
}

Grant SELECT with that filter attached and the analyst group sees UK rows without those two columns, through Athena, Redshift Spectrum or EMR, with no view to maintain and no duplicated table.

Now the limitation nobody mentions until it fails an audit. AWS states plainly that "filters apply only to read operations. Therefore, you can grant only the SELECT Lake Formation permission with filters". Filters govern what the query engines return. They do not change the object in S3, so anyone holding direct s3:GetObject on the prefix reads the raw Parquet and every filter becomes irrelevant. Fine-grained access is the second layer. The first is that the value is not in the file at all.

Scale decides how much this matters. On an Azure lakehouse serving 1,000+ business users across ten-plus banking sources, one unmasked column is not a single exposure, it is a thousand. That arithmetic is why these controls get bought at sector level, in banking and financial services and healthcare and the public sector alike, and it sits on your half of the shared responsibility split.

delete_forever Right to erasure in an immutable object store

This is the genuinely hard part, and I would rather say so than pretend a diagram solves it.

Article 17(1) gives a data subject "the right to obtain from the controller the erasure of personal data concerning him or her without undue delay". S3 objects are immutable, and one Parquet file holds thousands of rows belonging to thousands of people. There is no DELETE against a byte range, so the only native operation is "rewrite the whole file", which on a partitioned lake can mean rewriting many files to remove one row.

Controls deployed for other regulators actively fight you here. S3 Object Lock in compliance mode means "a protected object version can't be overwritten or deleted by any user, including the root user in your AWS account", and AWS is blunt about the escape hatch: "the only way to delete an object under the compliance mode before its retention date expires is to delete the associated AWS account." A WORM policy written for financial record-keeping and an erasure request are in direct conflict, and lawyers resolve that, not engineers.

Three implementable approaches, none of them free:

Table formats with row-level deletes. Iceberg or Hudi turn the problem back into SQL. Athena supports DELETE on Iceberg tables, and the documentation is explicit: "The UPDATE and DELETE statements follow the Iceberg format v2 row-level position delete specification", and "Athena SQL does not currently support the copy-on-write approach". Merge-on-read hides the row behind a delete file rather than removing it, and older snapshots still hold it. Expiring snapshots and compacting is the step that actually erases the bytes, and the step teams forget.

Crypto-shredding. Encrypt each subject's identifying fields under a per-subject data key, then destroy the key when the request arrives. The ciphertext stays put and becomes unreadable, which suits a lake full of immutable Parquet. Whether destroying a key counts as erasure is a legal question, so get it answered in writing first.

Layout for deletion. Partition or cluster so one subject's rows are concentrated rather than scattered across every file. Rewriting three files is a job. Rewriting thirty thousand is an outage.

A deletion is not done until it has reached backups, Data Catalog statistics, downstream extracts, BI caches and old snapshots. Article 17(2) is unusually pragmatic here, requiring "reasonable steps, including technical measures" while "taking account of available technology and the cost of implementation". Documented, reasoned, imperfect coverage beats an undocumented claim of completeness.

checklist The ingest sequence, in order

1. Drop first. Remove every source field with no stated downstream use.
2. Land raw into a short-lived bucket, one-day lifecycle rule, no human principals in its policy.
3. Classify each surviving sensitive column into redact, hash, tokenise or encrypt, and record the decision in the pipeline repo.
4. Apply the transform in one job that is the only writer to the raw zone: peppered HMAC for join keys, envelope encryption for recoverable fields.
5. Grant reads through Lake Formation data filters, never through direct S3 access to the underlying prefixes.
6. Run a full-sample detection job on a schedule, alerting on any column your config does not know about.
7. Enable CloudTrail data events on the landing and raw prefixes, delivered somewhere the audited account cannot edit.

Steps 1 and 7 get skipped most often, and they are the two cheapest on the list.

quiz Frequently asked questions

flag The pattern that holds up

The teams that get this right are not the ones with the cleverest cryptography. They are the ones who made a single job the only writer into the lake, wrote a per-column decision down next to the code, and treated deletion as a storage-layout question at design time instead of an incident eighteen months later.

All of that is engineering. Whether it is sufficient is a judgement for your data protection officer or your counsel, and the order matters: build the controls, document what they do and do not cover, then ask. If you are looking at a lake built to move data rather than account for it, and you need the retrofit sequenced so it does not break every downstream report, get in touch.