RAG - lesson 2 | HackIndia Skills

RAG - lesson 2 | HackIndia Skills | HackIndia

Example

Chunk format

Save chunks as JSONL. Each line is one searchable chunk with text and metadata.

Metadata is what lets you show citations later. Keep it boring and consistent.

{"chunk_id":"placement_rules_2024__p03__c01","source":"placement_rules_2024.pdf","page":3,"section":"Eligibility","text":"Students with one active backlog are not eligible for Day 1 companies unless the company policy allows it."}
{"chunk_id":"placement_rules_2024__p04__c02","source":"placement_rules_2024.pdf","page":4,"section":"Offer rules","text":"A student who accepts an offer through campus placement cannot sit for lower CTC companies later."}

Example

Small extraction script

This script extracts text from text-based PDFs using pypdf. It also handles .txt and .md files.

After running it, open the output files and inspect them. Do not trust extraction blindly.

# pip install pypdf
from pathlib import Path
from pypdf import PdfReader

RAW = Path("data/raw")
OUT = Path("data/processed_text")
OUT.mkdir(parents=True, exist_ok=True)

for path in RAW.iterdir():
    if path.suffix.lower() == ".pdf":
        reader = PdfReader(str(path))
        parts = []
        for i, page in enumerate(reader.pages, start=1):
            text = page.extract_text() or ""
            parts.append(f"\n\n[PAGE {i}]\n{text}")
        (OUT / f"{path.stem}.txt").write_text("".join(parts), encoding="utf-8")

    elif path.suffix.lower() in [".txt", ".md"]:
        text = path.read_text(encoding="utf-8")
        (OUT / f"{path.stem}.txt").write_text(text, encoding="utf-8")

print("done")

Prepare documents for search

Clean text wins

What to prepare

Folder layout

Chunk format

Extract the text

Small extraction script

Chunk by meaning

Avoid giant chunks

Do this now

Quick check