Blog/AI Security

AI supply chain security: checking model provenance before you deploy

Build an approval process for models, datasets and AI libraries so you know where every component came from and whether it has been tampered with.

By Sagar Makker · 7 min read · October 2026
Cctv Camera
In short
Every model, dataset and AI library you pull in is third-party code and needs vetting.
Verify source, integrity and licence, and prefer file formats that cannot run code.
Keep an AI bill of materials so you can answer what you run and where it came from.
The problem

This is for CTOs, platform leads and security heads whose teams download open models, embedding models, datasets and AI libraries to build features. Engineers often pull a promising model from a public hub and ship it within days. That component may have an unclear origin, a restrictive licence, or a file format that can execute code when loaded. This post gives you a lightweight approval process so every AI component has a known source, verified integrity and a named owner.

A data science team finds an embedding model on a public hub that performs well on Hindi and English text. They download it from a personal account with few details, load it in a notebook, and move it into production. Nobody checks who published it, whether the file was modified, or whether its format can run arbitrary code during loading.

Six months later, a customer's procurement team asks for a list of AI components, their sources and licences. Nobody has one. Different teams use different copies of similar models, a few with licences that restrict commercial use. When a library used for model serving announces a security fix, the team cannot quickly tell which services depend on it.

Why it happens
01Models are downloaded from unverified publishers with no review step
02File integrity is not checked against a published hash or signature
03Formats that can execute code on load are used without restriction
04Licences and usage terms for models and datasets are not recorded
05No inventory of AI components, so updates and fixes are hard to track
How to fix it
1
Create an AI component intake form
The platform lead introduces a short form for any new model, dataset or AI library: source, publisher, version, licence, intended use and owner. Nothing moves beyond a sandbox until the form is complete and reviewed.
2
Vet publisher and licence
Security and legal check that the publisher is identifiable and reputable, the model card describes training data and limits, and the licence allows your intended commercial use. Check the current position with counsel where terms are unclear.
3
Verify integrity and use safe formats
Engineering verifies downloaded files against published hashes or signatures and prefers weight formats that store tensors only. Formats that can execute code on load are blocked by default, with exceptions approved by security.
4
Mirror approved components internally
The platform team stores approved models and packages in an internal registry with pinned versions. Production pulls only from that registry, never directly from public hubs, so what was reviewed is exactly what runs.
5
Maintain an AI bill of materials
The component owner records each approved item in an AI bill of materials alongside the services that use it. Security subscribes to advisories and reviews the list on a fixed schedule for updates and fixes.
Worked example
A healthcare software firm's model register

A healthcare software company in Hyderabad used several open models for document classification and medical text extraction, each downloaded by different engineers over a year. A hospital client's IT team asked for a component list with sources and licences during contract renewal. The CTO set up an intake form, had security and legal review every model in use, replaced two with clearer licensing, converted weights to a tensor-only format, and moved all approved files into an internal registry with pinned versions. After the change, the team answered the client's questionnaire with a clear AI bill of materials, new models went through a short review, and a later library security advisory was mapped to affected services within the same day.

Common mistakes
×
Judging a model only by benchmark scores
Strong benchmark results say nothing about origin, licence or tampering. Vet the publisher, model card and licence first, then evaluate performance on your own data.
×
Pulling models from public hubs at runtime
A changed upstream file can silently alter production behaviour. Mirror approved, pinned versions in an internal registry and deploy only from there.
×
Forgetting datasets and evaluation data
Datasets carry licence, consent and poisoning risks too. Put them through the same intake, record their sources, and check personal data use with your data protection lead.
Where an AI agent helps
Input
Intake forms, model cards, licence text, file hashes and service dependency lists
Agent does
Checks publisher details, hashes, formats and licence flags, and updates the AI bill of materials
Human gate
Security and legal approve each component before it leaves the sandbox
Outcome
An approved component register and alerts for advisories on components in use
Checklist
Require an intake form for every AI component
Verify file hashes or signatures before use
Block code-executing model formats by default
Record licence and permitted use for each model
Serve production models from an internal registry
Review the AI bill of materials on a set schedule
What to measure
Components with completed intake review
Models served from internal registry
Components with unknown or unclear licence
Time to map an advisory to services
Questions
What is an AI bill of materials?

It is an inventory of AI components in your products: models, datasets, embedding models and key libraries, with source, version, licence, owner and the services using each. It lets you answer customer questions and respond quickly to advisories.

Are hosted model APIs part of the supply chain too?

Yes. Record the provider, model version, data handling terms and region for each hosted API you call. Track version changes, because a provider update can change behaviour even when your code does not.

Why do model file formats matter?

Some older formats can execute code when a model is loaded, so a tampered file could run commands on your server. Tensor-only formats store weights without executable content, which removes that risk during loading.

Who should approve a new model?

The engineering owner confirms technical fit, security checks source and integrity, and legal confirms the licence allows your use. For models touching personal or health data, include your data protection lead as well.

Keep reading
All posts →
Computer CodeAI Security
A practical AI security checklist before you launch an AI feature
5 min readRead →
Cctv CameraAI Security
LLM output validation: check AI answers before they touch your systems
7 min readRead →
Cctv CameraAI Security
Prompt injection explained for business leaders, and how to reduce risk
5 min readRead →
Next step

Want this fixed in your business?

Send the workflow. You get a scope and a fixed price within two working days — or a straight answer that it isn't a fit.

Secure by default · find → fix → verify · authorised scope only
Send a brief →enquiry@xdqlabs.com