What makes PII detection so hard?

As an AI researcher with a background in linguistics, I always like to understand more than the bare facts about what the latest models are capable of – I like to dig deeper into the properties of language that define the limits of current technology.

The problem of PII detection is a particularly interesting one because, on the face of it, PII is easy to define and easy for a human to spot, but not so easy to describe linguistically and sometimes hard for AI to grapple with.

What is PII?

PII is Personally Identifiable Information: data, typically text, which is connected to a specific individual and can be used to identify them, such as a name, address or phone number.

It’s useful to distinguish PII from sensitive data. PII can be one kind of sensitive data, but sensitive data more generally is any data that carries some risk if exposed, such as confidential financial information. In some situations, PII may also not be treated as sensitive, such as in minutes of public meetings.

There are grey areas in what counts as PII versus sensitive data. For example, specific details of a medical history may not identify a person directly, but a combination of details – like age, hospital and diagnosis – might be enough to identify someone. These are called quasi-identifiers.

In this article, I’ll mostly be considering clear-cut cases of PII.

The language of PII

Let’s think through names as the canonical example of PII. In the most straightforward cases, names are easy to describe and detect – both for humans and AI – because they have a bunch of predictable properties in English.

They’re structurally rigid, meaning that they usually occur in the same grammatical positions in sentences. As proper nouns, they behave like nouns, agree with the third-person, and can be replaced by pronouns like ‘he’, ‘she’ and ‘they’.

They’re semantically opaque, in that they can’t be interpreted from the context they appear in. If you don’t know what ‘ball’ means, you can make a good guess from “the golfer hit the ball,” but you can’t guess who Michael is from “the golfer hit Michael.” Similarly, regular nouns can often be interpreted from the way they look – you can figure out what a ‘golfer’ is if you know what ‘golf’ is – but name meanings are normally historical curiosities that are not readily apparent.

They’re lexically unambiguous, in that they usually don’t overlap with other words that look or sound the same but have different meanings. While you could get confused between a golf ball and a Yule ball, Michael is almost always a person.

They’re orthographically marked, which is a technical way of saying that they stand out in the middle of sentences in English as one of the few things that are capitalised.

And they have collocation patterns, which is also a technical term for the fact that names often occur alongside each other, as first names followed by middle names or surnames.

The confounds

What makes PII difficult sometimes is that, although all the above is easy enough to describe, names don’t have any of these properties uniquely or necessarily.

All of the structural properties of personal names are shared by entities like organisations and locations, which may or may not constitute sensitive PII depending on the circumstances. Names can also occur in atypical structures, like when a person is mentioned as an adjective (“that’s a very Michael thing to do”).

Likewise, organisations and locations can be semantically opaque, while many names could be confused as meaningful by a low quality AI model – someone with the surname Ball, for example, could be confused in context for a ball.

Overlapping with these possible confusions, some names are not lexically unambiguous. This is the case for Ball (or Black or Eve and so on), but names can also overlap with other named entities that may not be sensitive – for example, W. H. Smith might be a business rather than a person.

Depending on the quality of a data source, or the processing it may have undergone before some downstream analytics task, standard orthography may be lost and names may no longer be capitalised.

And also depending on the dataset, names may not occur alongside surnames to facilitate detection by collocation patterns, or some names may actually be mononyms, like Beyoncé (this could be by convention or legally).

The ramifications for AI detection

What this means altogether varies enormously depending on the dataset and the task intended for it.

For example, if an engineer has the benefit of a dataset analysis which confirms that all names are capitalised and the data pertains to medical histories (rather than, say, business reports), they might use very simple string-matching techniques to look for fixed linguistic patterns that they can describe in hand-written code.

However, as datasets get larger and more unpredictable, such simple techniques no longer suffice because they’ll start to yield false positives and false negatives due to the confounds we just explored.

The key to overcoming these difficulties is usually some kind of context awareness, which means understanding not just the formal patterns of the words, but the human intention behind them.

For a long time, Named Entity Recognition (NER) models have been a lightweight and high-performing standard for name identification, especially as they usually classify names separately from organisations and locations, so that users can customise the entities they wish to redact.

However, NER models are not perfect and they lack granularity, so where risk tolerance is low or the target PII is complex, they may lack the required sophistication.

The landscape has changed enormously since the popularisation of LLMs in the past few years, though, as the methods that are used to train LLMs were specifically designed for them to develop a kind of context awareness lacking in other ML models like traditional NER classifiers.

This means that LLMs have an impressive ability to respond naturally to questions about the likelihood of data being PII, but they still have weaknesses. The most obvious are that they’re general-purpose models whose cost far exceeds the typical needs of PII detection, and that they remain brittle where datasets contain information that are well outside the LLM’s training data.

For these reasons, many detection systems these days are actually multi-layered – instead of choosing between string-matching, NER or LLMs, engineers often select each for their strengths to cover the others’ weaknesses.