Today we dive into the world of model security. The first order of the day is to separate this into three areas of concern.
Each of these three has a different attacker profile with different objectives.
Which, at the end of the day, means you need to have controls in place that address each of them.
Let’s get into it.
1. Theft
In this kind of attack, an attacker is trying to get to your weights.
If they can get to them, they can make a copy for themselves and now have access to the proprietary information and potentially personal information about your customers that you encoded the weights with.
There’s nothing too glamourous about this attack path either – it comes down to insider risk or credential compromise. Here someone has or gains access to your artifact registry and just has to make a copy of a large binary file for themselves.
What stops it: access control on your artifact store, egress monitoring on large file transfers, encryption at rest with keys held separately, and having a good grasp of all your artifact locations
Why it gets missed: DLP tooling is tuned for documents and structured data. DLP’s aren’t usually configured to spot multi-gigabyte binaries being copied from a storage bucket.
2. Tampering
In this kind of attack, an attacker is trying to change your model in some way.
For regulated entities, this is a huge concern, as backdoored models will look like they behave correctly against all your tests, but then change in some expected behaviour when your attacker triggers them to.
For this attack path, the attacker is going through your supply chain. Examples can include a compromised base model from a public hub, a poisoned fine-tuning dataset, write access to the artifact registry, or an adapter swapped between approval and deployment.
What stops it: cryptographic signing at production and verification at deployment, integrity checks on the write path, and controlled provenance for every base model you consume.
Why it gets missed: there’s no signal. A tampered model file is the same size, loads normally, and passes functional testing. Detection isn’t available as a scanning problem, so the control has to be integrity rather than inspection.
3. Replication
This is an attack that isn’t reliant on unauthorised access.
For this attack path, an attacked with legitimate API access queries your model systematically, records the outputs, and trains their own model on those input-output pairs. Given enough queries, they get a model that approximates yours. Because there was no unauthorised access, every request gets authenticated, authorised, and logged as normal traffic
The security team is unlikely to detect this – simply because there was no anomalous activity in the first place.
What stops it: rate limiting, query pattern analysis, and restricting output detail such as returning class labels instead of full confidence distributions.
Why it gets missed: it can look like normal customer behaviour.
Taking stock of where you are today
Review your current inventory of controls, and if needed run tests against them for each of the three attack paths.
Then note the gaps. You may some controls in place but not the others.
For example, you’ve addressed theft by implementing access control on your artifact store. But you’re still vulnerable to tampering and replication.
Or you’ve addressed tampering by implementing signing. But that doesn’t stop anyone copying a signed model or stop anyone querying the API.
Or you’ve addressed replication by implementing rate limiting. I’m sure you get the point.
You should also extend this to your security vendors – if one says that their product secures your model, dig further into how and what kind of controls they have in place.
How do you prioritise
If you’re an Australian regulated entity, my starting position is to address them in order of: tampering, theft, replication.
Why?
I have tampering first, because it results in incorrect decisions being made about or for customers while you’re still seeing green on your control monitoring dashboard. And the controls you need you already have defined and implemented elsewhere as part of your software supply chain risk.
Then I’d address theft, because though the potential consequences of this are serious, it’s not as obvious to your customers if it does happen. And you likely also have controls in place elsewhere in other contexts.
I’d address replication last (unless you’ve built something genuinely proprietary). In the case of an organisation consuming a commercial foundation model, an attacker replicating your deployment is just getting a copy of something they could have licensed.
This ordering shifts if you’re a model producer. For everyone else, it’s tampering that connects most to your prudential obligations.