When a team ships a retrieval-augmented generation (RAG) system, the first question is usually "how accurate is it?" — but accuracy is the wrong lens. A confident answer that cites the wrong passage is worse than a hedged one that doesn't, and a single score buries that distinction.
The metric we lean on instead is faithfulness: how much of the answer is actually supported by the retrieved context.
Faithfulness, precisely
If an answer makes \(n\) claims and \(k\) of them are grounded in the retrieved passages, its faithfulness is simply
So a response with \(k = 9\) supported claims out of \(n = 10\) scores \(F = 0.9\). The one ungrounded claim is exactly where hallucination hides — and it's invisible to a top-line accuracy number.
Why averaging across queries lies
Reporting the mean faithfulness \(\bar{F} = \frac{1}{m}\sum_{i=1}^{m} F_i\) over \(m\) queries feels reasonable, but it smooths over the tail. A system that is perfect on easy lookups and unfaithful on the multi-hop questions that matter can still post a healthy average. We look at the distribution — especially the worst decile — not just the mean.
What we actually do
- Decompose every answer into atomic claims, then check each against the retrieved context.
- Separate retrieval quality from generation quality — a faithful answer over the wrong context is luck, not skill.
- Weight by decision stakes, so the questions that drive real choices count for more.
Faithfulness measures whether the model earned its answer from the evidence — not whether the answer happened to be right.
A quick sketch of the per-answer check:
def faithfulness(claims, context):
supported = [c for c in claims if entailed_by(c, context)]
return len(supported) / len(claims)
It's a humble metric, but it's honest — and honesty is the whole point when a model's output feeds a real decision.