OOmnistu

Blog / AI Development

AI Development

Data and model poisoning explained: what it means for teams using AI models

Understand how poisoned training or retrieval data can affect AI behavior and which safeguards product teams should discuss.

OOmnistu
Reviewed 2026-11-16 · 2 min read

OWASP lists Data and Model Poisoning as LLM04 in its Top 10 for LLM Applications (2025). We read the project's page in full on 15 November 2026, and not the references. This article summarises that page. Most small teams will not train a model from scratch, but the page applies to anyone who fine-tunes, adds their own documents (embeddings), or downloads a model from a shared repository.

What it is

Poisoning is manipulating pre-training, fine-tuning or embedding data to introduce vulnerabilities, backdoors or bias. OWASP calls it an integrity attack. It can target pre-training, fine-tuning, embedding and transfer learning. Risks are higher with external data that may be unverified or malicious. Possible results: weaker performance, biased or toxic output, and exploitation of downstream systems.

Two further points from the page:

WHY A BACKDOOR IS HARD TO TEST FOR - THE PAGE'S "SLEEPER AGENT" POINT The same model one set of weights, two behaviours ORDINARY INPUTS No trigger present what tests usually feed it Behaves normally passes tests that omit the trigger A TRIGGER APPEARS The trigger input planted by the attacker Hidden behaviour wakes the page's scenario words: authentication bypass, data exfiltration, hidden commands WHY A BACKDOOR IS HARD TO TEST FOR THE PAGE'S "SLEEPER AGENT" POINT The same model one set of weights, two behaviours Ordinary inputs A trigger appears Behaves normally passes tests without the trigger Hidden behaviour wakes auth bypass, data exfiltration, hidden commands Tests without the trigger may miss the backdoor - hence the page's detection warning
From the OWASP LLM04 page: a backdoor may leave behaviour normal until a trigger appears - a "sleeper agent" the page says is hard to test for and detect. The trigger outcomes shown are the page's own scenario wording (authentication bypass, data exfiltration, hidden command execution). The mitigations listed above - provenance tracking, red-teaming, robustness testing and monitoring - are the countermeasures.

Mitigations listed

  1. Track data origins and changes (OWASP CycloneDX or ML-BOM) and verify data at every development stage.
  2. Vet data vendors and validate model outputs against trusted sources.
  3. Sandbox the model away from unverified data sources and use anomaly detection.
  4. Use specific datasets for each fine-tuning use case.
  5. Use infrastructure controls so the model cannot reach unintended data.
  6. Version data (the page names DVC) to detect manipulation.
  7. Keep user-supplied information in a vector database so it can change without retraining.
  8. Red-team the model and test robustness.
  9. Monitor training loss and model behaviour against thresholds.
  10. Use retrieval-augmented generation and grounding at inference where appropriate. These do not make untrusted data safe: vector storage and RAG still need provenance, permissions and poisoning checks; neither is a universal defense.

Scenarios on the page

Biased output from manipulated data; unfiltered toxic data; falsified documents created by a competitor or other bad actor; misleading data inserted through prompt injection; and a backdoor trigger that could allow authentication bypass, data exfiltration or hidden command execution.

What a small team can ask

Related: AI agent integration risks, RAG explained for founders, prompt injection explained.

Limits

One OWASP page. References, MITRE ATLAS and NIST links unread. We did not check any named tool. No statistics or vendor claims. Which items apply depends on whether you train, fine-tune or only call a hosted model, and the page does not draw that line.

By Omnistu.

Sources

Previous
Adding an AI agent to your app: the "excessive agency" risk and how to limit it