The choice to buy, fine-tune or build an AI model is often made in an architecture forum.
They likely included considerations related to cost, capability and time to market. They likely didn’t include considerations related to the evidence each option will require once a model is in production.
I raise this now, because ignoring the latter set of considerations create very different assurance problems for each of the buy, fine-tune or build options.
Defining our terms first, by “buy” I mean consuming a managed model through an API. By “Fine-tune” I mean adapting a third-party base model using your own data. By “Build” I mean training new model weights from the beginning.
From a boundary standpoint, a fine-tuned model can still be hosted by the provider, and a model built internally can still depend heavily on external data, libraries and cloud infrastructure.
With that understood, here’s how each choice changes the evidence you need to produce for assurance.
Option 1: Buy – this still carries accountability
When you consume a model through an API, the provider manages the base-model training, model weights and serving infrastructure.
You avoid the need to operate most of the ML platform but remain responsible for the way the model is selected, configured, connected and used.
So the main assurance requirement you carry is your dependency itself.
Why is that?
- You cannot inspect your provider’s complete training corpus or reproduce their training process.
- Evaluation is largely limited to documentation supplied by your provider and testing you can perform through the service interface.
- If the provider changes the model, their safety controls or the surrounding infrastructure, you may have a system change on your hands even though your own code has not.
Your evidence therefore can come from two sources.
The first is what was defined as part of your supplier relationship. Think: change-notification obligations, incident reporting, data-handling terms, audit rights, service levels and support for exit.
The second is from your operational processes. Think: pre-deployment evaluation, output monitoring, incident records and repeated testing against the risks that matter to your use case.
An easy way to think about it:
- Your contract with the provider tells you what the provider has agreed to do.
- Your monitoring tells you what your system is doing.
At a portfolio level, there is a concentration consideration to factor in In its April 2026 letter, APRA reported that some entities were heavily dependent on a single provider across multiple AI use cases. Few had demonstrated credible contingency planning or tested their ability to exit or substitute a critical AI provider.
So “buying” a model once might be an efficient play. What happens though when you make that model the default across the organisation, and thus you concentrate your risk?
Option 2: Fine-tuning – adds a second assurance boundary
Fine-tuning starts with a base model developed by someone else and adapts it using additional training data.
But your base-model dependency remains. You still need evidence about the provider, the original model, its known limitations and changes to the version on which your model depends.
You then add responsibilities of your own.
You need to know where the fine-tuning data came from, whether it was permitted for that use, how it was prepared and which version entered the training job. You need records of the training configuration, evaluation results, resulting artefact and every later change.
If you run the fine-tuning environment, you also take responsibility for its access controls, dependencies, data movement, logging and artefact integrity. If the provider runs it for you, those responsibilities become supplier-assurance questions. They do not disappear.
This is why fine-tuning is frequently underestimated. The word sounds like a small adjustment. The assurance boundary can expand considerably.
There is also a privacy issue, although it needs to be described accurately.
Fine-tuning data is not simply copied into a model as a set of readable records. Training changes the model’s parameters. Models can nevertheless memorise or reproduce elements of their training data, particularly where records are duplicated, distinctive or overrepresented.
Deleting a person’s record from the original dataset does not automatically remove its influence from a model that has already been trained.
That poses tough questions when information must be corrected, destroyed or excluded from further use. Depending on how the model was adapted, the response might involve replacing an adapter, retraining from an earlier checkpoint, applying an unlearning technique, restricting outputs or retiring the artefact. Full retraining is sometimes the cleanest answer, but it is not the only technically possible one.
Australia does not currently provide a general individual right to erasure equivalent to the GDPR. APP 11.2 does, however, require reasonable steps to destroy or de-identify personal information when it is no longer needed for a permitted purpose, while APP 13 deals with correction. Overseas laws and contractual commitments may create further obligations.
Your most practical control is provenance. If you cannot identify which data was used to produce which model version, you cannot make a defensible decision about what needs to be corrected, retrained or withdrawn.
Option 3: Build – gives you control (provided you keep the records)
Training from scratch removes the dependency on a third-party base model. It does not make the system independent of everything else.
You may still rely on external datasets, open-source libraries, cloud infrastructure, specialist hardware and third-party labelling services. Internal development also requires independent validation. The team that built the model should not be the sole judge of whether it is safe and fit for purpose.
What building gives you is the opportunity to create a complete evidence chain.
You can record the data sources, preprocessing, training code, dependencies, configurations, evaluation results and artefact lineage. You can retain the exact version assessed before deployment and trace later changes back to an approved training run.
That evidence exists only if the operating model was designed to produce it. Owning the training environment does not help when experiments were run through undocumented notebooks, datasets were assembled without provenance or model files were copied outside the registry.
The cost is substantial. You need the data pipeline, controlled training environment, compute capacity, artefact management, specialist staff, evaluation capability and ongoing monitoring. You must sustain them for as long as the model remains in use.
For most Australian enterprises, training a general-purpose foundation model from scratch would make little economic sense. Building can be justified for narrower models where proprietary data creates a real advantage, suitable external models do not exist and the capability is important enough to operate over the long term.
But fine-tuning isn’t always your highest-risk option
It would be hasty to generalise and say that fine-tuning always carries more total risk than buying or building.
Risk depends on the use case, data, model, deployment arrangement and consequences of failure. A small adapter trained on controlled data for a low-impact internal task is not inherently riskier than a foundation model trained from scratch.
Your more defensible conclusion is this:
“Fine-tuning can create your broadest assurance boundary because it combines dependence on an external base model with responsibility for an internal training process.”
That is the part architecture forums often miss.
Buy usually has the lowest internal engineering burden and the greatest supplier dependency.
Fine-tune adds data provenance, training governance and artefact management while retaining some of that dependency.
Build creates the largest operating burden, but gives the organisation greater control over the evidence if the lifecycle is properly recorded.
None of these answers is automatically wrong. What matters is whether your assurance cost has been priced alongside your engineering cost.
Ask for evidence before approving an architecture
Before an architectural decision is formally signed off, ask your teams:
“If APRA asked us tomorrow how this model was built, changed, tested and monitored, what evidence could we produce without waiting for one of our vendors to get back to us?”
For a bought model, the answer will rely heavily on supplier evidence and your own behavioural monitoring.
For a fine-tuned model, you will need both the supplier evidence for the base model and your records for the adaptation.
For a model trained from scratch, most of the evidence should come from your own systems, assuming you created and retained it.
An architecture decision determines where your evidence must come from. But it should also determine who is accountable for obtaining that evidence before the model reaches production.
Sources