When an AI feature gets it wrong: misinformation and overreliance for product teams
Review misinformation and overreliance risks when people use AI output to answer questions or make decisions.
OWASP lists Misinformation as LLM09 in its Top 10 for LLM Applications (2025). We read the project's page in full on 15 November 2026. We did not read its references, so the examples below describe risks rather than independently verified incident reports.
What it is
Misinformation is false or misleading output that looks credible. A main cause is hallucination: the model fills gaps with statistical patterns and produces content that seems accurate but is made up. The page says bias in training data and incomplete information also contribute. Overreliance makes it worse: users trust the output and do not verify it, then build it into decisions.
Forms it takes, per the page
- Factual inaccuracies. A model can give travellers incorrect policy information; verify the actual policy before making a decision.
- Unsupported claims. A chatbot can fabricate legal citations; check the original legal materials with a qualified reviewer.
- Misrepresented expertise. A chatbot can make unsupported health claims sound like expert knowledge or turn settled evidence into apparent debate. Health decisions need qualified sources and professional review.
- Unsafe code. The model may suggest insecure or non-existent libraries.
What OWASP recommends
- Retrieval-augmented generation, to ground answers in verified sources (see our RAG article).
- Fine-tuning or embeddings, and techniques such as chain-of-thought prompting.
- Cross-checking and human oversight for critical content, with reviewers trained not to over-trust the AI.
- Automatic validation of key outputs in high-stakes settings.
- Telling users about the risks and limits.
- Secure coding practices for code the model suggests.
- Interface design that labels AI content, adds filters and states the intended field of use.
- User training on limits and independent verification.
Scenarios from the page
- Attackers find package names that a coding assistant often invents, publish malicious packages with those names, and developers install them.
- A company deploys a medical-diagnosis chatbot without enough accuracy checks. This is a hypothetical risk scenario: no attacker is needed for inaccurate output to cause harm.
What this means for a feature you are planning
- Decide what an incorrect answer would cost in your product, and add checks or human review in proportion.
- Show users where answers come from and that they can be wrong.
- Verify any package or dependency a model suggests before installing it.
Related: RAG explained for founders, prompt injection explained, LLM output handling.
Limits
One OWASP page. References unread; examples are not incident reports; no statistics, no vendor claims. The page does not say how much each mitigation helps, and neither do we.
By Omnistu.