How do masking, tokenisation and pseudonymisation change an LLM prompt?

The US National Institute of Standards and Technology (NIST) treats de-identification as an umbrella term for any process that removes the link between identifying data and the person it describes. There is not one technique but a collection of approaches and tools, and each has a different level of effectiveness.

• Masking swaps a sensitive value for a generic type label and is irreversible.
• Tokenisation swaps a sensitive value for a meaningless surrogate and records the pair in a mapping, keeping the “token” so that it can be restored. Two different people stay as two different tokens, and the model’s answer can be turned back into real names afterwards.
• Pseudonymisation swaps in a plausible value of the same kind, one that keeps whatever properties the task needs to reason over. This stays reversible for as long as the mapping exists.

Which method is best depends on what the prompt gets from the entity, and on whether altered data needs to be traced back to a real person or account after the prompt.

What survives the transformation and what doesn’t

The same NIST publication separates direct identifiers, such as names and Social Security numbers, from quasi-identifiers, which only identify someone when combined with other information. It also warns that tools which simply mask personal information do not always do enough to count as de-identification, as individuals could still be identified from indirect information.

Prompts raise a problem that transforming whole datasets doesn’t. A transformed prompt goes into a reasoning step, and whoever reads the output usually needs the real names back to make sense of the answer. When buying a tool, it’s important to check whether the model still has what it needs to answer, whether separate entities are still distinguishable, and whether the response can be restored reliably.

Redaction costs some tasks more than others

A team at Amazon Web Services used spaCy’s named entity recognition (NER) to find entities in public benchmarks, replaced each one with its type token, and compared model scores on the original and redacted versions. Sentiment analysis and entailment (testing whether a model can link texts together and find the nature and direction of that link) barely suffered, typically losing only 1–5%. Comprehension question answering, however, lost more than 25%.

The first two tasks test model ability on wider, more general aspects of texts that focus on overall tone or direction, but when the task gets more specific, like comprehension, redacting information has a hugely negative effect.

Another useful finding is how far a model’s results swing between tasks. Claude 3.5 Sonnet scored 92.8 on the BBH date-understanding task with nothing redacted, but 40.6 with the full entity set redacted. Redacting names alone brought it back to 86.3. These were chain-of-thought prompts with few-shot examples taken from the unredacted training set, so the results describe those benchmarks under those specific conditions and nothing broader.

It’s worth planning for the disparity in models’ treatment of redacted text. Mistral 7B filled the redacted name tags with sequential placeholder values of its own and then reasoned over them, and on a few redacted sets this scored higher than the original text did. Claude and GPT-4o pointed out that information was missing and gave no answer. A model that quietly invents stand-ins for your masked entities won’t fail visibly in testing, so it is important to keep this in mind when choosing a scrubbing method.

One renewals prompt, three ways to hide

Running one prompt through each method shows exactly what each one changes.

A renewals team sends this prompt to a hosted model:

Masking turns both customers into the same person label and both renewal dates into the same date label. The dates lose their order as well as their differentiation, so the model has nothing to rank the accounts by.

Tokenisation fixes the ambiguity by giving each entity its own value. Presidio’s pseudonymisation sample illustrates this as a custom operator: in its example sentence about Peter, Heidi and Nicole, each name becomes its own numbered person token, and the client keeps an entity mapping dictionary that it uses to reverse the substitution.

In the renewals prompt, the two customers stay distinct and with each downstream task the value stays with the right person. However, the dates are still a problem, because opaque date tokens can’t be compared. They need a rule of their own.

Pseudonymisation leaves the prompt reading as ordinary prose. The names are swapped for other plausible names, and the real dates are either kept or shifted by the same offset. Shifting preserves both their order and the six-week gap between 3 March and 14 April. Once the original names are restored, the account manager has an answer they can actually use.

Whoever holds the mapping holds the names

Masking has no mapping to protect, but tokenisation and pseudonymisation both depend on one. Whoever holds it can re-identify the text, so it is vital to keep this secure.

Presidio’s sample comes with a blunt warning: its logic is not thread-safe and can give wrong results when run concurrently, because the mapping has to be shared across threads, workers or processes. If you plan to put a transformation step in a shared gateway that handles many prompts at once, load-test that path. The sample’s de-anonymiser raises an error when a token isn’t in the mapping for its entity type, so a failed restoration shows up as an error rather than as a stray token in an answer a user reads.

Nakato’s workflow, Reversible Semantic Pseudonymisation (RSP), employs a context-aware pseudonymisation process. It detects sensitive information inside the customer’s environment, replaces them with semantic twins that are chosen to keep the properties the task needs, sends only the transformed text to the model, and restores the original references in the response. Detection runs on small models inside that environment, and the mapping stays there; only the transformed data leaves.

For more information on deployment and RSP, see Architecture and How It Works.

Evaluate the tools yourself

When choosing a privacy vendor, build a set of real prompts with known correct answers, and score each one under four conditions: untransformed, masked, tokenised and pseudonymised. Fix the metric before you start. NIST makes the general point that de-identified data accurate for one purpose may not be accurate for another, and prompts are no different.

Next, run an entity ablation. In the AWS limited-redaction experiment, leaving task-critical entity types unredacted recovered most of the lost accuracy on the worst-hit tasks; tracking seven shuffled objects went from 4.8 back to 93 once names were left in. An ablation is the quickest way to find out which of your entities need to keep a usable value. Then test restoration under realistic concurrency and count the tokens that fail to map back.

The risks that change, and when to re-test

None of this removes residual risk. NIST quotes guidance that even when properly applied, de-identification leaves some risk of identification – very small but not zero – and cites the Evidence Act’s point that a data asset posing no privacy risk on its own may pose one once combined with other available information.

It’s also important to re-test whenever the model or its version changes, since Mistral 7B and Claude 3.5 Sonnet behaved very differently on identical redacted inputs. Re-test as well when a prompt template starts carrying new entity types or much longer context. Even when you have a process you are happy with, changing any property can affect performance.