OOmnistu

Blog / AI Development

AI Development

When an AI feature gets it wrong: misinformation and overreliance for product teams

Review misinformation and overreliance risks when people use AI output to answer questions or make decisions.

OOmnistu
Reviewed 2026-11-16 · 2 min read

OWASP lists Misinformation as LLM09 in its Top 10 for LLM Applications (2025). We read the project's page in full on 15 November 2026. We did not read its references, so the examples below describe risks rather than independently verified incident reports.

What it is

Misinformation is false or misleading output that looks credible. A main cause is hallucination: the model fills gaps with statistical patterns and produces content that seems accurate but is made up. The page says bias in training data and incomplete information also contribute. Overreliance makes it worse: users trust the output and do not verify it, then build it into decisions.

Forms it takes, per the page

  1. Factual inaccuracies. A model can give travellers incorrect policy information; verify the actual policy before making a decision.
  2. Unsupported claims. A chatbot can fabricate legal citations; check the original legal materials with a qualified reviewer.
  3. Misrepresented expertise. A chatbot can make unsupported health claims sound like expert knowledge or turn settled evidence into apparent debate. Health decisions need qualified sources and professional review.
  4. Unsafe code. The model may suggest insecure or non-existent libraries.

What OWASP recommends

Scenarios from the page

THE PACKAGE-NAME SCENARIO FROM THE OWASP PAGE 1. Coding assistant invents plausible package names (hallucination) 2. Attackers find the invented names and publish malicious packages under them 3. Developers install the suggested package without verifying 4. Result the project installs a malicious package THE PACKAGE-NAME SCENARIO FROM THE OWASP PAGE 1. Coding assistant invents plausible package names 2. Attackers find the invented names and publish malicious packages under them 3. Developers install the suggested package 4. Result: the project installs a malicious package
One of the OWASP LLM09 page's two scenarios, quoted in the list above: attackers find package names that a coding assistant often invents, publish malicious packages with those names, and developers install them. The page presents it as a risk scenario, not a verified incident; the checks and verification habits in the next section are the countermeasure.

What this means for a feature you are planning

Related: RAG explained for founders, prompt injection explained, LLM output handling.

Limits

One OWASP page. References unread; examples are not incident reports; no statistics, no vendor claims. The page does not say how much each mitigation helps, and neither do we.

By Omnistu.

Sources

Previous
Runaway AI usage: unbounded consumption, denial of wallet and how to cap them