How good are LLMs at PII detection?

The current landscape

The automatic identification of Personally Identifiable Information (PII) is a classic AI task which is especially valuable to users who need to identify and protect large quantities of sensitive data.

The oldest approach to PII detection actually pre-dates modern machine learning and uses simple string-matching techniques to look for tell-tale signs of PII, like known vocabularies of names or businesses.

These techniques are still the preferred method for extremely rigid data types, like the characteristic number patterns of credit cards, but for more flexible data types, they’ve generally been superseded by machine learning approaches.

For a long time, Named Entity Recognition (NER) classifiers have been the go-to tool. Most typically, these are models which are trained on labelled datasets to predict whether a piece of data is a person, organisation, location or similar category.

NER classifiers are robust tools which are still in use, but there has been much excitement about the advent of LLMs, since they have a more nuanced understanding of language and therefore potentially fewer problems with more complex data. But how much is this actually borne out in technical analysis?

Overview of the data

Because LLMs are a relatively new technology and they have so many potential applications, there aren’t a large number of studies dedicated to analysing their ability at NER.

However, a paper published this year by Dzung Pham and colleagues introduced a new benchmark, AmBench, specifically focused on the capabilities of LLMs for PII detection.

AmBench probes two potential failure modes in particular:

• named regularity bias, which is where detection fails despite contextual cues that should make the task easy, e.g. a name occurs in the vicinity of a pronoun but it still isn’t detected as a name
• benign prompt injection, which is where the data includes text which may read like a prompt and could lead to names being leaked in a response if the model doesn’t understand that it shouldn’t repeat them

AmBench has a particularly tight focus on rare names that are one edit distance away from a different category of proper nouns (e.g. Albanir vs. Albania), which facilitates a robust analysis of one particular kind of problem, but we need to be restrained about the significance of the benchmark for LLM capabilities more broadly.

With that in mind, the authors’ results were as follows:

The authors summarise that recall on ambiguous names is generally far worse than on popular names, with up to a ~40% drop off with GPT-5 and 4o.

This is actually similar to the performance level of human judges on the same data, but that’s rarely a consolation for a pass rate of only 60%. This also suggests that LLMs can run into difficulties even with relatively simple tasks because of biases in their learning process.

Another paper from this year by Terra Blevins and colleagues introduced the UNER v2 benchmark, targeting entity recognition across multiple languages, and it reported the following results on small- and medium-sized LLMs:

The F1 scores are a combined performance metric which the authors don’t break down in detail, but the important facts they summarise is that, compared to human annotators, LLMs have a marked bias towards false positives when identifying names. This is further corroborated by another recent paper by Qi Zhan et al. that pitted LLMs against traditional NER classifiers.

In some scenarios, this tendency towards false positives may actually be considered a useful feature of the models, since false negatives (i.e. PII which goes undetected) are usually much more harmful.

The developing landscape

Although there are a handful of papers that explore the generic capabilities of LLMs for PII tasks, there’s still scant literature describing their specific failure modes.

This is partly to be expected because it’s difficult in principle to make a benchmark that can represent the vast variety of NER and PII detection tasks that are of interest to different types of users.

However, this underscores a general lesson about LLMs in the current moment: although they’ve been designed as general-purpose tools and can often get the bulk of a job done with minimal effort, they have a jagged intelligence with unpredictable weaknesses, and it’s essential for engineers to understand the linguistic properties of the datasets they’re working with before they can judge whether an LLM is the right tool for their task.

Most often, in fact, they’ll probably want to complement an LLM with other tried and tested tools, like string-matching and classifiers.