Swapping names isn’t enough: getting pseudonymised data approved for an LLM

If you want to send personal data to a large language model (LLM), it has to be altered to remove identifying details. As explored in other articles, there are different options for de-identifying data. Nakato uses pseudonymisation. Names, account numbers, and other identifiers are swapped for contextually equivalent stand-ins, and the key that links them back is kept somewhere secure. Getting this pseudonymisation pipeline approved by a risk or data protection committee is a separate step.

Under UK GDPR, the committee needs written evidence on seven points, and it helps to start from one fact: pseudonymised data is still personal data for as long as you hold the key. The committee isn’t approving data that has stopped being personal; it’s approving a measured reduction in risk.

What the law actually says

The ICO uses the UK GDPR definition that data is pseudonymised when it can no longer be linked to a specific person without additional information, and that information is kept separately and protected. In plain terms, one dataset goes in and two come out: the pseudonymised text, and the key (often a mapping table) that turns it back into the original.

Things get trickier when the text goes to an externally hosted LLM. Without the key, it may be anonymous in the model provider’s hands, but the ICO’s draft guidance warns that you can’t assume so. You have to weigh what else the recipient knows or could find in public, the time, money and technology it would take to identify someone, and the controls on the data once it arrives. The ICO’s guidance is also under review following the Data (Use and Access) Act, so record which version you relied on and when.

Seven pieces of evidence needed, in the expected order

Under UK GDPR, the organisation that decides how and why personal data is used is called the controller. The ICO expects the controller to set goals, assess risks, choose a technique, decide who applies it, and document all of it, typically in a data protection impact assessment (DPIA). The evidence pack can follow the same order:

• A goal statement. Who, specifically, should be unable to identify people?
• A threat list: insiders with access, in your organisation or at a processor, and outsiders trying to learn more from the transformed text.
• The technique and its settings, with the reasoning that links them to those threats.
• Named roles for who runs the transformation, who can reach the key and who handles the output.
• An identifiability check on a real sample of transformed prompts, read as the recipient would read them.
• Accuracy evidence that the transformed text still does the job.
• A retesting plan, since the ICO expects techniques to be evaluated regularly, not once.

Prose talks: why free text is harder

The techniques the ICO lists – hashing, encryption and tokenisation – all deal with identifiers. For example, a name or policy number becomes a scrambled value or a random token. Prompts are usually prose, though, and prose carries wider clues. A complaint letter can mention a rare job or a local event that points to one person after every name has gone. Details like these are called quasi-identifiers. The ICO advises considering whether rare attributes like these can be removed.

In a study published at ICLR 2024, ETH Zurich researchers asked GPT-4 to infer personal details from 520 Reddit profiles, then ran the text through Azure Language Service, a commercial anonymisation tool that replaced detected names, places, dates and other entities with asterisks. GPT-4 still guessed the author’s location correctly about 55% of the time, down from about 86%.

This is why the leak rate under active attack is an important benchmark. It demonstrates how easily a person can be re-identified when a transformed prompt is attacked, showing how much quasi-identifiers leak about an individual. This is also why we’d make the identifiability check a condition of approval: it shows how much these clues give away about a person, before anyone external sees the text.

A worked example

Say an insurer wants to use an externally hosted LLM to summarise complaint letters. The letters will sit on the model provider’s systems, so the goal is to make sure nobody who could access them there can identify complainants. The named threats are prompts kept in the provider’s logs, someone stealing the mapping table, and someone inferring an identity from the narrative. Names, policy numbers, addresses and dates of birth become tokens, and the mapping table sits in a separate database that only two named engineers at the insurer can reach.

A reviewer reads forty transformed letters as an outsider with access to the provider’s copy would. Three describe a fire at a named factory in a small town, where one worker was hurt, and give the month. The worker’s name is gone, but anyone who follows the local news could work out who it was. So the employer, town and incident type are swapped for substitutes that keep what the summary needs: a workplace injury claim, delayed by a given number of weeks.

Handlers then compare summaries of original and transformed letters for any lost fact they’d have acted on. The committee approves summarisation and nothing else. A later idea, using complaint histories to decide how to treat individual complainants, needs its own assessment. This is because the ICO treats general analysis, like spotting trends across many people, differently from analysis aimed at taking action about a specific person. Pseudonymisation doesn’t make a new purpose acceptable on its own; the new use needs its own assessment.

What to write into the approval

One legal point belongs in the approval. Section 171 of the Data Protection Act 2018 makes it an offence to knowingly or recklessly re-identify de-identified data without the controller’s consent. Identifiability testing counts as deliberate re-identification, so record the consenting controller and the authorised scope first.

If prompts are pseudonymised as they’re sent, the approval should name where that happens, such as a gateway that requests pass through, and list which applications route through it. Anything that bypasses it sends the original text.

Finally, attach a review trigger: a new model provider, new routing or new prompt templates should each send a fresh sample through the identifiability check, because each changes what leaves your network or who receives it.