One question to ask before you approve an AI program

If you’re submitting a board paper for an AI program, there’s likely a general structure you’ve been used to, something like:

1. Use case
2. Expected benefit
3. Risk section
4. Controls summary
5. Recommendation

This may span 12-15 pages and present a very compelling case by the team members who are looking for the board to say “yes”.

On the flip side, if you’re on the board and reviewing this paper, it means that as a director you face a unique set of challenges:

1. Everything in the paper looks plausible
2. Nobody in the board room can independently verify the technical claims
3. The cost of asking a naive question in front of executive management is higher than most people are willing to admit

So here’s one question a director can ask that doesn’t require any technical knowledge and can’t be addressed with a single slide:

“Can we produce, on request, the complete list of data sources this system relies on? And can we prove that this list is complete?”

And then just wait for the response.

Why does this work?

“On request”

You didn’t ask “do we have it documented somewhere”. Nor did you ask “could we assemble it”.

You asked for it “on request”, meaning that it exists now, in a form someone can hand over to you.

This is important because that’s the form a regulator will ask for it. And there’s a difference in compliance positions between having something and being able to produce something.

“Complete list”

The key word here is “complete”.

Most teams can produce a list. But they struggle to assert that nothing is missing, for example that there isn’t an additional data source that’s now connected to the system, or that there’s now a document store the retrieval layer reaches that wasn’t in the original design.

A partial list isn’t an answer your regulator will want to see. A system with 12 known data sources and an unknown number of unknown ones has the same assurance position as a system with none.

“Data sources it relies on”

This statement is deliberate in being broader than just your training data.

If the system does retrieval, the live document stores are data sources. If the system calls a tool that returns information, that’s also a data source. If the system reads from a vector database that’s been stood up recently, that’s also a data source.

The answer you’re listening for is whether they mention anything beyond the training set. If they don’t, that’s an alarm bell to follow.

“Prove that list is complete”

This is the statement that separates a real capability from just slideware.

Proof means something structural. For example, network-level egress controls that constrain what the system can reach would count. Provenance recorded at ingestion rather than reconstructed later would count too. An architecture where connecting a new data source requires passing through a gate you control would also count.

But “we asked the team and they told us” doesn’t count as proof. Even if it comes with a signature on it.

What makes this question so powerful

There’s three reasons why:

1. It’s binary. Your get back a yes or no, and everyone in the room hears it. Most AI risk questions produce answers that sound like yes without being yes, but this one is hard to fudge in front of an audience.
2. It’s upstream of nearly everything else you’ll want to have accounted for. Poisoning, privacy transparency, prudential assurance, critical operations mapping, incident forensics. Every one of those depends on knowing what the system reads. Answer this and you’re most of the way to answering several others.
3. It’s answerable before the program starts, which is when a board still has leverage. Ask this question at an approval gate and you’ll be shaping the build. Ask this question down the road and you’ll likely be commissioning a remediation project.

The follow-up questions, if you hear a confident “yes”

1. Who has write access to those sources, and which of them can someone outside the organisation contribute to?
2. When a data source is added, what has to happen? Also, make sure to name the gate for it.
3. Which of these sources have we reviewed against the purpose the data was originally collected for?

I covered that there’s a clock that’s for being able to answer that last question in yesterday’s post.

What a good answer sounds like

Any director just hearing back “yes, we have full lineage” should be concerned.

A good answer sounds more like: “we have complete provenance for the training corpus and our primary retrieval stores, and this covers our decisioning path. We don’t yet have it for the following secondary sources – here they are, here’s why, and here’s the date we will.”

That sets you up to be an organisation that has looked in the right place….

…meaning a confident, unqualified “yes” almost always reflects an organisation that hasn’t.

---

Aaannd that’s a wrap for week two. Next week I’m diving into the training environment itself: orchestration, dependencies, the build process, and why the ML supply chain is a problem your AppSec team has likely already solved.