A Taxonomy of LLM Hallucinations in Financial Analysis

"Hallucination" is used as a catch-all term for LLM errors. But not all hallucinations are equal. Some are obvious and harmless. Others are subtle and can cost you money.

After reviewing outputs from financial analyses, we've identified five distinct levels. The levels are ordered by how much domain expertise you need to catch them.

The Five Levels

Level 1 hallucinations are the easiest to catch. The model invents a source that doesn't exist. You can verify by checking whether the document is real.

We see this most often when models try to be "helpful" by providing specific citations they weren't trained on. Perplexity has an advantage here — it actually searches and cites, rather than fabricating.

Level 2 is wrong numbers. The document exists, but the model misreads or invents the figures. This is harder to catch because you need to actually read the source.

Common failure modes:

• Confusing revenue with contribution ex-TAC (we saw this with Criteo)
• Using TTM numbers when quarterly was asked for
• Mixing up fiscal years with calendar years
• Getting the sign wrong on growth rates

Level 3 is where things get dangerous. The number is correct, but crucial context is missing. The statement is technically true but misleading.

Level 3 hallucinations require you to know what context is missing. If you don't already understand payment processor accounting, you won't catch this.

Level 4 is faulty reasoning that sounds correct. The model cites real numbers and makes an inference that seems logical but is actually wrong.

We built the "Treadmill Test" specifically to catch this. Many companies with high buyback yields are actually treading water or going backwards when you account for dilution.

Level 5 is the worst. The model understands the surface facts but misunderstands the structural dynamics of the business or industry.

In the Criteo example: yes, Retail Media is growing. But we calculated that 13% of revenue growing at 20% cannot mathematically offset 87% of revenue declining at 5%. The "pivot" narrative is true in direction but false in magnitude.

Level 5 errors are nearly impossible to detect — short of developing deep familiarity yourself. And here's the uncomfortable truth: even "experts" get structural calls wrong. The market is full of professional analysts who missed obvious-in-hindsight structural shifts.

Defense by Level

How Our Framework Addresses Each Level

• L1 (Fake Citations): We direct models to use aggregators over SEC EDGAR (aggregators are indexed better). Perplexity provides real-time citations.
• L2 (Wrong Numbers): We mandate cross-source reconciliation in Stage 0. If numbers don't match, flagged.
• L3 (Missing Context): We have specific checks for client funds, lease liabilities, SBC, and other commonly omitted items built into the prompt.
• L4 (Reasoning Errors): The Treadmill Test, NCI check, and other mandatory calculations force explicit math. Harder for models to hide bad logic.
• L5 (Structural Errors): Honestly? We can't reliably catch these either. We mark what we've manually checked vs. LLM-only, but structural understanding requires ongoing learning — not a checklist.

The Uncomfortable Truth

We can catch L1-L3 reliably with our current framework. L4 we catch most of the time with mandatory calculations. L5 is where we have to be honest: we can only catch structural errors in domains we understand.

If an LLM makes a structural error about oil drilling physics, we might miss it. If it makes a structural error about tech sector business models, we're more likely to catch it.

This is why we include a "What We Checked" section in every analysis. We want readers to know exactly what has been manually verified vs. what's still LLM-only — so you can decide what additional verification you need to do.