Builder session 7 · Answer correctness

Part 0

Grounded is not the same as correct

A Recall answer can be perfectly faithful to a retrieved chunk and still be wrong - because the chunk was wrong, or because the question needed information no chunk carried. Faithfulness grades an answer against the context it used. Correctness grades it against the answer it should have given. That second thing needs a reference: a human-written ground-truth answer, agreed in advance, exactly like the golden set from b1. Today we score against it two ways and blend them.

Live - presented in session Self-study - full depth after class ★ Try it now Sources covered at the end

★ What you build today A claim-level factual F1 scorer over a reference, a semantic-similarity score from embeddings, and the RAGAS answer-correctness call that blends them - the first metric in this track that measures whether Recall is right, not just consistent.

Part 1 · covers factual correctness by claim F1

Factual correctness by claim F1 9 min live

You cannot compare two paragraphs with a single equals sign - phrasing differs, order differs, a right answer can be shorter or longer than the reference. So RAGAS breaks both the response and the reference into atomic claims, then checks each claim against the other text with a natural-language-inference (NLI) step. That turns "are these two answers the same" into a countable set of true positives, false positives, and false negatives - and from those, an F1.

🔍 Click to zoom - factual correctness: decompose into claims, classify TP/FP/FN, compute F1

LiveWhat TP, FP, and FN mean here4 min▶

The three counts are the whole engine. Once you have them, precision, recall, and F1 fall out mechanically. The trick is reading each count as a specific kind of rightness or wrongness in the answer.

True positive (TP) = a claim in the response that is supported by the reference. This is the answer getting something right - a fact it said that the ground truth agrees with.
False positive (FP) = a claim in the response that the reference does not support. This is the answer adding something wrong or made up - it hurts precision.
False negative (FN) = a claim in the reference that the response failed to say. This is the answer leaving out something it should have covered - it hurts recall.

The F1 formula, once Precision = TP / (TP + FP) - of what the answer claimed, how much was right. Recall = TP / (TP + FN) - of what should have been said, how much was said. F1 = 2PR / (P + R) - the harmonic mean, so an answer must be both accurate and complete to score high. Say everything correct but omit half, and recall drags F1 down; pad with a wrong claim, and precision does.

Self-studyFactualCorrectness in RAGAS4 min read▶

RAGAS ships this as a metric. You give it the response and a reference, and it runs the decompose-then-NLI pipeline for you, returning the claim-level F1. Over a dataset with a reference column, it scores every row.

Python · RAGAS FactualCorrectness on one samplefrom ragas import evaluate from ragas.dataset_schema import SingleTurnSample from ragas.metrics import FactualCorrectness sample = SingleTurnSample( response="Refunds are issued within 5 business days to the original card.", reference="Approved refunds are returned to the original payment method within 5 business days.", ) scorer = FactualCorrectness() # claim decomposition + NLI -> claim F1 score = await scorer.single_turn_ascore(sample) print(score) # 0-1; higher means the response's claims match the reference's

Across Recall's whole golden set you pass a dataset that carries a reference for each question, and let evaluate() roll it up.

Python · FactualCorrectness over a dataset with referencefrom datasets import Dataset from ragas import evaluate from ragas.metrics import FactualCorrectness data = Dataset.from_dict({ "user_input": ["How long do refunds take?", "How do I reset my password?"], "response": [recall_answer(q) for q in questions], # what Recall produced "reference": ["Refunds reach the original method in 5 business days.", "Use the Forgot password link; a reset email arrives in minutes."], }) result = evaluate(data, metrics=[FactualCorrectness()]) print(result) # mean factual-correctness F1 across the set

Reference is mandatory here FactualCorrectness cannot run without a reference. There is nothing to compute TP, FP, and FN against otherwise. If your dataset has no ground-truth column, this metric silently has no meaning - which is exactly why b1's golden set matters so much.

Part 2 · covers semantic similarity + answer correctness

Semantic similarity and the blend 9 min live

Claim F1 is strict and structural. It can miss the case where an answer means the same thing but shares no claims cleanly - a paraphrase, a different framing. So RAGAS adds a second, softer view: embed the response and the reference, and take the cosine of the two vectors. That is semantic similarity, a 0-to-1 read of "do these two answers mean the same". Answer correctness then blends the strict claim F1 with this soft similarity into one number.

🔍 Click to zoom - semantic similarity via cosine, then the answer-correctness blend

LiveSemantic similarity: cosine of two embeddings3 min▶

Similarity ignores claims entirely. It asks a blunter question: embedded as vectors, do the response and the reference point the same way? Cosine of 1.0 means identical direction (same meaning); near 0 means unrelated.

It forgives phrasing. "Refunds take five days" and "You will get your money back within a business week" share few words but embed close. Claim F1 might penalise the wording; similarity does not.
It is a single soft number. No TP/FP/FN, just one cosine per pair, 0 to 1. Cheap once you have embeddings.
It can be fooled by topic. Two answers about refunds that disagree on the number of days can still score high on similarity - they are on-topic. That is why you do not use it alone.

Python · RAGAS SemanticSimilarityfrom ragas.dataset_schema import SingleTurnSample from ragas.metrics import SemanticSimilarity from ragas.embeddings import LangchainEmbeddingsWrapper from langchain_openai import OpenAIEmbeddings scorer = SemanticSimilarity( embeddings=LangchainEmbeddingsWrapper(OpenAIEmbeddings()) ) sample = SingleTurnSample( response="You get your money back within a business week.", reference="Refunds reach the original method in 5 business days.", ) score = await scorer.single_turn_ascore(sample) print(score) # cosine of the two embeddings, 0-1

LiveAnswer correctness = factual F1 blended with similarity4 min▶

Answer correctness is the composite. It takes the strict claim F1 (accuracy and completeness of facts) and the soft semantic similarity (meaning overlap) and combines them, weighted, into one 0-to-1 score. High answer correctness means the response is both factually aligned with the reference and reads as the same answer.

Correctness needs a reference; faithfulness does not. Use correctness when you have a known-right answer to grade against - regression sets, QA benchmarks, anything with ground truth. Use faithfulness (b5) when you only have the retrieved context and want to catch made-up claims, with no ground truth in hand.
Correctness = vs a known answer. Faithfulness = vs retrieved context. They can disagree loudly: an answer can be faithful to a bad chunk (grounded but wrong) or correct while drawing on context you did not track. Different references, different questions.

Python · RAGAS AnswerCorrectness over a datasetfrom datasets import Dataset from ragas import evaluate from ragas.metrics import answer_correctness data = Dataset.from_dict({ "question": questions, "answer": [recall_answer(q) for q in questions], "ground_truth": references, # your written-in-advance right answers }) result = evaluate(data, metrics=[answer_correctness]) print(result) # blends factual claim-F1 + semantic similarity into one 0-1 score

Real world A partially-correct answer Recall answers "Refunds are issued within 7 business days to your bank account." The reference says "within 5 business days to the original payment method." The "refund is issued" claim is a TP; "7 business days" and "to your bank account" are FPs; the reference's "5 business days" and "original payment method" become FNs. Claim F1 lands low - lots of wrong and missing claims. But semantic similarity stays high - both are clearly refund-timing answers on the same topic. Answer correctness lands in the middle: the blend refuses to call a wrong-number answer fully correct, but does not treat it as unrelated either. That middle score is the honest read of "close, but wrong on the details that matter".

Honesty note - the default weights are not published Answer correctness combines factual similarity (claim F1) and semantic similarity with a weighting, and the split is tunable. RAGAS does not publish a fixed default ratio we can quote, so do not memorise a number. What is stable: it is a weighted blend of those two components, both require a reference, and you can adjust the weights when you configure the metric. Treat the exact mix as an implementation detail, not a fact to recite.

Build-along · take it further

Write references, then reason about the scores ★ 12 min · pen and paper

You cannot grade correctness without ground truth, so first you author it - then you predict how a flawed answer scores under each lens before any tool runs.

Write three references. Pick three Recall questions from your golden set (refund timing, password reset, one of your own). For each, write the one-sentence answer you would accept as fully correct. These are your ground_truth entries.

Draft a partially-correct answer. For one question, write an answer that gets the topic right but changes one fact (a wrong number, a wrong channel). This is your test case.

Reason on factual F1. List the claims in your answer and in your reference. Mark each TP, FP, or FN. Estimate precision, recall, F1. It should be low - the wrong fact is an FP and the missing right fact is an FN.

Reason on semantic similarity. Would the two answers embed close? Almost certainly - same topic, same shape. Note that similarity stays high while F1 drops, and write one line on why answer correctness (the blend) is more trustworthy than either alone.

★ Recall's correctness after b7 Recall can now be graded against ground truth, not just its own context. You can compute claim-level factual F1, semantic similarity, and the answer-correctness blend over the golden set - and you know that a right-topic, wrong-fact answer lands in the honest middle. In b8 we turn all of these scores into a pass/fail gate that blocks a bad change from shipping.

Homework

Before session b8 ◐ 45 min total

Add a reference column to your b6 RAGAS dataset for at least five Recall questions, then run FactualCorrectness and read which rows score lowest.
Run answer_correctness on the same set and compare its ranking of the answers to the factual-F1 ranking - note where the blend disagrees with strict F1 and ask why.
Take one answer that is faithful (high b5 faithfulness) but wrong, and confirm its answer correctness is low - proof the two metrics measure different things.
Read: RAGAS docs on factual correctness, answer correctness, and semantic similarity.

Source material

Official sources covered

Taught from official docs. This page covers the reference-based correctness metrics in RAGAS - the claim-F1 mechanic, semantic similarity, and their blend.

✓RAGAS · Factual Correctness (claim decomposition + NLI + F1)Part 1 · TP/FP/FN over claims, P/R/F1; needs reference

✓RAGAS · Answer Correctness (weighted blend)Part 2 · factual F1 + semantic similarity; weights tunable, default not quoted

✓RAGAS · Semantic Similarity (cosine of embeddings)Part 2 · response vs reference embedding, 0-1

Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · In factual correctness, a claim the response makes that the reference does NOT support is a...

A response claim unsupported by the reference is a false positive - the answer added something wrong. FPs sit in the denominator of precision = TP/(TP+FP), so they drag precision down.

2 · Semantic similarity in RAGAS is computed as...

Semantic similarity embeds both texts and takes the cosine of the two vectors - a 0-to-1 read of shared meaning that forgives phrasing differences.

3 · You have no ground-truth answers, only the retrieved context. Which metric applies?

Correctness metrics (factual correctness, answer correctness, semantic similarity) all require a reference. Without ground truth, use faithfulness, which grades the answer against the retrieved context instead.

Builder session 7 cheat sheet · pin this

Correctness vs faithfulnessCorrectness grades vs a known-right answer; faithfulness grades vs the retrieved context. Different references.

Reference is mandatoryAll three correctness metrics need a ground-truth answer written in advance. No reference, no meaning.

Claim decompositionResponse and reference are each split into atomic claims, then compared - not matched as raw paragraphs.

TP / FP / FNTP = response claim supported by reference. FP = response claim it does not support. FN = reference claim the response missed.

Claim F1Precision = TP/(TP+FP), Recall = TP/(TP+FN), F1 = 2PR/(P+R). Must be accurate AND complete.

Semantic similarityCosine of response vs reference embedding, 0-1. Forgives phrasing; can be fooled by same-topic disagreement.

Answer correctnessWeighted blend of claim F1 + semantic similarity. Weights tunable; the default split is not published - do not quote one.

Right topic, wrong factLands mid-score: low claim F1, high similarity. The blend calls it "close but wrong", which is honest.