Data and model poisoning explained: what it means for teams using AI models
Understand how poisoned training or retrieval data can affect AI behavior and which safeguards product teams should discuss.
OWASP lists Data and Model Poisoning as LLM04 in its Top 10 for LLM Applications (2025). We read the project's page in full on 15 November 2026, and not the references. This article summarises that page. Most small teams will not train a model from scratch, but the page applies to anyone who fine-tunes, adds their own documents (embeddings), or downloads a model from a shared repository.
What it is
Poisoning is manipulating pre-training, fine-tuning or embedding data to introduce vulnerabilities, backdoors or bias. OWASP calls it an integrity attack. It can target pre-training, fine-tuning, embedding and transfer learning. Risks are higher with external data that may be unverified or malicious. Possible results: weaker performance, biased or toxic output, and exploitation of downstream systems.
Two further points from the page:
- Models shared through open repositories can carry risks beyond poisoned data, such as malware hidden through "malicious pickling", which can run harmful code when the model is loaded.
- A backdoor may leave behaviour normal until a trigger appears, which the page says is hard to test for and detect (a "sleeper agent").
Mitigations listed
- Track data origins and changes (OWASP CycloneDX or ML-BOM) and verify data at every development stage.
- Vet data vendors and validate model outputs against trusted sources.
- Sandbox the model away from unverified data sources and use anomaly detection.
- Use specific datasets for each fine-tuning use case.
- Use infrastructure controls so the model cannot reach unintended data.
- Version data (the page names DVC) to detect manipulation.
- Keep user-supplied information in a vector database so it can change without retraining.
- Red-team the model and test robustness.
- Monitor training loss and model behaviour against thresholds.
- Use retrieval-augmented generation and grounding at inference where appropriate. These do not make untrusted data safe: vector storage and RAG still need provenance, permissions and poisoning checks; neither is a universal defense.
Scenarios on the page
Biased output from manipulated data; unfiltered toxic data; falsified documents created by a competitor or other bad actor; misleading data inserted through prompt injection; and a backdoor trigger that could allow authentication bypass, data exfiltration or hidden command execution.
What a small team can ask
- Where does each dataset and each downloaded model come from, and who could have changed it?
- Is the file format one that can execute code when loaded?
- Can we see what changed in our data between versions?
- If we fine-tune on user content, who can add to it?
Related: AI agent integration risks, RAG explained for founders, prompt injection explained.
Limits
One OWASP page. References, MITRE ATLAS and NIST links unread. We did not check any named tool. No statistics or vendor claims. Which items apply depends on whether you train, fine-tune or only call a hosted model, and the page does not draw that line.
By Omnistu.