RAG - lesson 6 | HackIndia Skills

RAG - lesson 6 | HackIndia Skills | HackIndia

Example

Eval file example

Keep the format boring. It should be easy for a recruiter or teammate to read.

[
  {
    "id": "q1",
    "question": "What is the refund deadline for the workshop?",
    "expected_source": "sample_docs/workshop_policy.md",
    "expected_evidence": "Refund requests must be submitted at least 7 days before the workshop date."
  },
  {
    "id": "q2",
    "question": "Does the policy mention hostel accommodation?",
    "expected_source": null,
    "expected_evidence": "Not answerable from the provided documents."
  }
]

Example

Tiny evaluator script

This script does not judge truth by itself. It creates a file where you can add pass or fail after reading the cited evidence.

import json
from pathlib import Path

# Replace this import with your app function.
# It should return: {"answer": "...", "citations": [{"source": "...", "text": "..."}]}
from app import answer_question

questions = json.loads(Path("eval_questions.json").read_text())
results = []

for item in questions:
    output = answer_question(item["question"])
    results.append({
        "id": item["id"],
        "question": item["question"],
        "answer": output["answer"],
        "citations": output["citations"],
        "expected_source": item["expected_source"],
        "expected_evidence": item["expected_evidence"],
        "grounded": "TODO: pass/fail",
        "notes": "TODO: explain using cited text"
    })

Path("grounding_eval.json").write_text(json.dumps(results, indent=2))
print("Wrote grounding_eval.json")

Measure grounded answers

One idea

What to measure

Create a test set

Eval file example

Run the check

Tiny evaluator script

Common mistake

Use AI carefully

Do this now

Quick check