Four controls to isolate and protect your ML platform

So you just asked your application security team to inspect your ML platform.

Now your question is, what should they look for?

Start by treating your model training as a software build process…

…It consumes code, dependencies and data, then produces an artefact…

…An artefact that may be deployed into production.

You already now from my past posts that a compromised training environment can expose its data, alter the model being produced or provide access to other systems connected to it.

So what controls should you have in place for these risks?

Here are the four controls I recommend that you prioritise:

1. Separate experimentation from controlled training

Experimentation is interactive and unpredictable. A data scientist may install a new package, download a model from a public repository, run unfamiliar code or change a notebook as part of their day.

A production training pipeline should operate differently. It should run reviewed code against identified datasets, use approved dependencies and produce reproducible outputs through a controlled process.

Many ML platforms still combine these activities because the platform began as an analytics environment and not a build environment. That used to be ok when the analytics were kept locally. But it’s now becomes a problem when its outputs are promoted into customer-facing or operational systems.

How I recommend you ring fence:

• Experimental environments can access approved research data and external resources within defined limits.
• Production training runs through automated pipelines using pinned code and dependencies.
• Experimental users cannot write directly to the approved model registry.
• Moving work from experimentation into the training pipeline requires a controlled promotion step.

This allows researchers to work without giving every notebook a direct route into production.

2. Control what enters and leaves the training network

Training environments often have broad outbound internet access because they need packages, base models and datasets. This same level of access gives malicious code an attack path where it can send stolen data or establish command-and-control communications.

A safer approach is to route external dependencies through managed internal services. This way your training environment retrieves packages, models and approved datasets from internal mirrors or proxies. Those services handle external downloads, scanning, version retention and policy enforcement.

For example:

• Python packages come from an internal package repository.
• Base models are admitted through a model-ingestion process.
• External datasets are downloaded into a quarantine area before approval.
• General internet access from production training jobs is denied or restricted to specific destinations.

Any conversion of an untrusted pickle-based model should also occur in quarantine. Converting a model to safetensors may first require reading the unsafe source file, so the conversion process needs disposable compute, no credentials and tightly restricted network access.

This design gives your organisation a record of what entered the environment and removes most reasons for training jobs to communicate directly with the internet.

3. Protect the model artefact path

Access control on a model registry tells you who was permitted to write to it. It does not establish that the model being deployed came from the approved training run or remained unchanged afterwards.

The training pipeline should calculate a digest for each output and sign the artefact, or an attestation referring to it, when the run completes successfully. Promotion and deployment processes should then verify that signature before accepting the model.

The associated record should identify:

• the model digest and version;
• the training code commit;
• the dataset versions or snapshots;
• the dependency and container versions;
• the pipeline that produced the artefact; and
• the approvals applied before promotion.

This provides a chain from an approved training run to the model deployed in production. It also supports incident investigation, reproducibility and the evidence behind governance or regulatory disclosures. The Privacy Act’s automated-decision transparency requirements do not prescribe this technical lineage record, but producing reliable disclosures becomes considerably harder when the organisation cannot establish which data, code and model version underpin a system.

Registry permissions should follow the same model. Training pipelines may submit candidates, but only the promotion process should be able to mark an artefact as approved for production.

4. Match compute isolation to the sensitivity of the workload

Shared cloud infrastructure is appropriate for many training workloads. The decision changes when the environment processes sensitive personal information, valuable proprietary data or models whose weights represent a significant commercial asset.

For those workloads, ask the provider for specific information about the isolation boundary:

• Is the GPU dedicated to one customer or shared?
• Can the underlying host be shared with other tenants?
• How is GPU and host memory sanitised before the hardware is reassigned?
• What contractual or independent assurance supports those claims?
• Which administrators can access the host, workload or attached storage?

“Dedicated instance” can mean different things across services, so confirm whether the commitment applies to the virtual machine, the GPU, the physical host or all three.

Dedicated capacity costs more and may reduce scheduling flexibility. It should therefore follow the organisation’s data classification and threat model. A team training on public data may reasonably accept shared infrastructure. A workload using health records or detailed financial information warrants a stronger isolation decision and evidence that the chosen service provides it.

Where HSMs fit

Hardware security modules are useful here, although their role is narrower than some AI security discussions suggest.

An HSM can protect the keys used to encrypt training data and model artefacts. It can also protect the signing key used to establish that an approved pipeline produced a particular model.

The model weights themselves do not live in the HSM. They are too large, and the training or inference process needs to operate on them in accelerator memory. Protecting models while they are in use requires controls such as workload isolation, confidential computing where supported, restricted administrative access and hardened runtime environments.

Use the HSM for key custody and signing. Do not treat its presence as evidence that the model is protected throughout training or inference.

Implement them in this order

For most organisations, I would sequence your next steps as follows:

1. Separate experimental activity from controlled training and promotion.
2. Introduce managed dependency sources and restrict outbound access.
3. Sign or attest approved artefacts and verify them before deployment.
4. Adopt dedicated compute where the sensitivity of the data or model justifies it.

But this exact order may change if there you face an immediate exposure. An organisation already training on sensitive customer data over shared infrastructure may need to address compute isolation sooner. Likewise, a registry that allows broad write access more pressing remediation.

These controls should normally sit within the existing platform and software supply-chain security programme. The ML-specific ask is to apply them to notebooks, datasets, training jobs, model formats, registries and accelerators.

That gives your organisation an accountable owner, an established delivery mechanism and controls that operate consistently across conventional software and machine learning enviroments.