There was a time when the default deterrent against data poisoning was the use of scale.
The prevailing wisdom was that while poisoning is theoretically possible, to shift a model trained on hundreds of billions of tokens, you’d need to control a meaningful percentage of the training corpus.
And you can’t…
Nobody can…
Because the dataset is too big to poison…
Then October 2025 came about.
The study that questioned this
In a collaboration between Anthropic’s Alignment Science team, the UK AI Security Institute and the Alan Turing Institute ran the largest pretraining poisoning experiments that had been done to date. They used models from 600 million to 13 billion parameters, trained from scratch on Chinchilla-optimal datasets ranging from 6 billion to 260 billion tokens.
Their finding challenged the prevailing wisrom of the time:
In turned out what determines whether the attack works isn’t the percentage of the corpus that’s poisoned. It’s the raw number of poisoned documents. And that number is near-constant regardless of how big the model or the dataset gets.
It’s around 250 documents.
They found 100 documents wasn’t enough.
But at 250, the backdoor took hold consistently across every model size tested.
For the 13B model, 250 documents worked out to roughly 0.00016% of training tokens. The same 250 documents worked on the 600M model, where they represented 0.0035%.
More than 20 times the clean data.
Same handful of poisoned files.
Same result.
What this meant
Think about what scaling normally does for an attacker.
Normally, a bigger target costs more to attack. More data means proportionally more poison. The defender’s growth imposes cost on the attacker.
Here it doesn’t.
This means that while the attack surface, the public internet, keeps growing….
The attacker’s workload stays flat at a few hundred documents…
Which means the economics move toward the attacker as models get bigger.
And 250 documents is not a hard thing to produce. The Turing Institute’s own framing was that it would be relatively easy to create 250 poisoned Wikipedia articles.
The gotchas and fine print
This is where most coverage of this study stopped.
The researchers were careful about what they had and hadn’t shown, and stripping those caveats out overstates their finding.
The backdoor tested was simple and low-stakes – a trigger keyword that made the model emit gibberish. A denial-of-service attack, essentially.
It’s still an open question whether the same near-constant dynamic holds for frontier-scale models beyond 13B parameters. And it’s an open question whether it holds for genuinely dangerous backdoors (the kind that bypass safety behaviour or emit vulnerable code) which are harder to install than “produce nonsense on this keyword”.
So the truth is that it’s not proven that 250 documents can make a frontier model write exploitable code.
But the load-bearing assumption underneath enterprise thinking (that dataset scale is itself a defence) is where the ground shifted. And that was the assumption most organisations were relying on implictly.
“We don’t pretrain, so this doesn’t apply to us”
A large chunk of Australian enterprises may say that they don’t pretrain anything right now. So the natural response is that this is a frontier lab problem.
Two reasons that’s wrong.
Firstly: the same study found the same dynamics during fine-tuning. And plenty of Australian organisations fine-tune – on support tickets, on claims histories, on internal documentation, on data with a much shorter path from an outsider’s keyboard to your training set than the public internet has.
Secondly: more pressing is that if you’re doing RAG, the number isn’t 250.
It’s closer to 1.
Yes, 1.
That’s because retrieval doesn’t require an attacker to shift the model’s weights. It requires them to get one document into a store that the retriever will surface for a given query. There’s no training run to survive, no gradient to influence, no dilution against a corpus. Write a document that ranks well for the query you care about, get it into the knowledge base, and the model will read it and act on it.
That’s the same category of attack…. with a radically lower cost.
And that’s the deployment pattern most enterprises have implemented.
Why you probably won’t detect it
Here’s the part that makes this an assurance problem rather than a tooling problem.
A poisoned document is not malformed. It contains no signature, no payload, no anomalous structure. It’s a normal-looking document that happens to teach an association you didn’t intend.
There is no scanner for that. There can’t really be – you’d be scanning for semantic intent in text that looks, by construction, unremarkable.
So detection has to come from somewhere else. And the options are fewer than you may think:
- Provenance. Not “is this document malicious” but “where did this come from, who could write to it, and can I reconstruct that chain”. This is the only control that scales, and it sits with the data governance capability.
- Behavioural testing against triggers you don’t know. Hard. Genuinely unsolved. You’re trying to find a keyword you can’t guess.
- Post-hoc model analysis. Emerging, promising, not yet a product category you can buy in Australia.
Which leaves provenance as the practical answer for the next couple of years. And provenance is exactly what a pipeline built for correctness rather than integrity was never designed to produce.
Next steps
Take your training and RAG data sources. Sort them into three buckets:
- Sources only your staff can write to.
- Sources your customers or partners can write to.
- Sources anyone on the internet can write to.
Many organisations may discover something in bucket three that they’d assumed was in bucket one. Like scraped content, public forums, syndicated feeds, a vendor dataset with unclear upstream provenance.
That third bucket is your real poisoning surface. Everything else in this post has been leading up to showing you how much smaller it needs to be than you thought ☺️.
Sources
- Souly et al., Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples, arXiv:2510.07192 — https://arxiv.org/abs/2510.07192
- Anthropic, A small number of samples can poison LLMs of any size — https://www.anthropic.com/research/small-samples-poison
- The Alan Turing Institute, LLMs may be more vulnerable to data poisoning than we thought — https://www.turing.ac.uk/blog/llms-may-be-more-vulnerable-data-poisoning-we-thought