RAG (Retrieval Augmented Generation) interview questions & answers

Think of an LLM as a brilliant student who took a closed-book exam — trained on a huge pile of internet text up to some cutoff date, and then that training stopped. It knows a lot of general facts, but it has never seen your company's HR manual, your private PDFs, or anything that happened after training. Ask it something outside that knowledge and it either says "I don't know" or, worse, confidently makes something up (hallucination).

In RAG, instead of relying purely on what the model memorized, you give it an open-book exam: before it answers, you go fetch the relevant pages from YOUR documents and hand them to the model along with the question. RAG stands for Retrieve → Augment → Generate: retrieve relevant chunks from your data, augment the prompt with them, then let the LLM generate an answer grounded in that context. The model's weights never change — you are only changing what goes into the prompt.

RAG has two separate phases. Indexing (offline, done once, or whenever documents change): load your documents → split into chunks → embed each chunk (Ch3) → store the vectors in a vector DB (Ch3 — ChromaDB/Pinecone/pgvector). Query (online, every single question): embed the user's question → retrieve the top-k nearest chunks from the vector DB → build a prompt containing those chunks as context plus the question → the LLM generates the final answer.

Retrieval alone isn't enough to stop hallucination — you also need a grounding prompt. This is a system instruction that tells the model: "Answer ONLY using the context below. If the answer is not in the context, say you don't have that information — do not guess." Pair that with citations: return which chunk(s) the answer came from, so a user (or an interviewer) can verify it. This grounding + citation combo is the anti-hallucination core of RAG, and it is exactly what makes a RAG answer trustworthy instead of a confident guess.

🌍 Real-world example: Ask a plain LLM "What is my company's annual leave policy?" and it will guess a generic number or admit it doesn't know. Feed it the actual HR policy text as context with a grounding prompt, and it correctly answers "24 days" — and can point to exactly which document that came from.

💡 RAG = Retrieval Augmented Generation — retrieve relevant text from your own data and paste it into the prompt so the LLM answers from facts it was never trained on.

💡 Indexing phase = the one-time (or whenever-data-changes) job of preparing your documents into searchable vectors — chunk, embed, store.

💡 Query phase = the per-question job of embedding the question, retrieving matching chunks, and asking the LLM to answer using them.

💡 Grounding prompt = an instruction that restricts the LLM to answer only from the given context, and to admit when the answer isn't there — the main defence against hallucination.

💡 Citations = telling the user which source chunk/document each part of the answer came from, so the answer is verifiable.

Standard definition: RAG (Retrieval Augmented Generation) is a technique where relevant text is retrieved from an external knowledge source and inserted into the LLM's prompt as context before generation, so the model answers using up-to-date or private data it was never trained on, without changing the model's weights.

from openai import OpenAI
import chromadb

client = OpenAI()
chroma = chromadb.PersistentClient(path="./rag_db")
collection = chroma.get_or_create_collection("company_docs")

# ============================
# INDEXING PHASE (run once)
# ============================
documents = [
    "Annual leave policy: Every employee gets 24 days of annual leave per year.",
    "Sick leave: Employees are entitled to 12 days of sick leave annually.",
    "Work from home: WFH is allowed 2 days per week with manager approval.",
]
collection.add(documents=documents, ids=[f"doc_{i}" for i in range(len(documents))])

# ============================
# QUERY PHASE (run per question)
# ============================
def ask(question):
    # RETRIEVE: nearest chunks from Ch3's vector search
    results = collection.query(query_texts=[question], n_results=2)
    chunks = results["documents"][0]

    # AUGMENT: grounding prompt + citations
    context = "\n".join(f"[{i+1}] {c}" for i, c in enumerate(chunks))
    prompt = f"""Answer ONLY using the context below. If the answer is not in
the context, say "I don't have this information in the documents."
Cite the source number(s) you used, like [1].

Context:
{context}

Question: {question}
Answer:"""

    # GENERATE: grounded answer
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
    )
    return response.choices[0].message.content

print(ask("How many annual leaves do I get?"))
print(ask("What is the CEO's name?"))

Think of an LLM like a new employee. There are two totally different ways to make them better at their job: hand them a reference folder they can flip through before answering any question, or send them on a training course that reshapes how they actually think and speak. RAG is the reference folder. Fine-tuning is the training course.

In Gen AI, RAG (Retrieval Augmented Generation, Ch4) retrieves relevant text from your documents at query time and pastes it into the prompt — the model's weights never change, only what it reads changes. Fine-tuning does the opposite: you retrain the model's weights on thousands of labelled examples so its underlying behaviour changes permanently — no extra documents needed at answer time.

The decision framework is simple once you ask ONE question: is the problem missing knowledge, or wrong behaviour?

• Missing knowledge ("the model doesn't know our refund policy / today's price list / this new law") → RAG. Facts that change often, need to be traceable to a source, and shouldn't require retraining every time a document changes.
• Wrong behaviour ("the model knows the facts but won't reply in our exact tone / JSON schema / medical-report format / rare domain jargon") → fine-tuning. Style, structure, and consistency that a prompt alone struggles to lock in every single time.

🌍 Real-world example: A hospital chatbot needs to (a) answer from THIS week's bed-availability sheet — that's RAG, because the data changes daily and every answer must be checkable against a source — and (b) always reply in a strict clinical-note format the hospital's compliance team signed off on — that's a candidate for fine-tuning, because the shape of every answer must be identical, not just the facts.

💡 Knowledge vs behaviour = the single most important lens: RAG injects facts into context; fine-tuning bakes a skill/style into the weights. 💡 Grounding / citations = RAG can point to the exact chunk/document a fact came from; a fine-tuned model cannot — once a fact is baked into weights there is no "source" to cite. 💡 Update speed = RAG updates the moment you re-index a document (minutes); fine-tuning needs a new labelled dataset and a retraining run (hours to days) to change even one fact.

Standard definition: RAG changes what the model reads (retrieved context injected into the prompt, weights untouched) to add or refresh knowledge cheaply and with citations; fine-tuning changes what the model is (weights retrained on labelled examples) to change behaviour, tone, or output format permanently — the two solve different problems and are often combined, and most real systems should start with a well-prompted RAG pipeline before ever considering fine-tuning.

Comparison table

Dimension	RAG	Fine-tuning
Fixes	Missing/changing knowledge	Wrong behaviour, tone, format
Cost	Low — embedding + vector DB	High — GPU training + MLOps
Update speed	Minutes (re-index a document)	Hours to days (retrain)
Citations / traceability	Yes — cite the exact source chunk	No — facts are baked into weights
Data needed	Your existing documents	Hundreds-thousands of labelled examples
Handles new/changing facts	Yes, natively	Poorly — needs retraining every time
Locks in consistent style/format	Weak — depends on prompt discipline	Strong — becomes the model's default
Hallucination risk	Lower (grounded in retrieved text)	Same as base model unless RAG added too

When to use neither

If the model already knows the facts AND already produces the right style, you may need nothing but better prompting — a clearer system prompt, a few-shot example, or a stricter output-format instruction. Don't reach for RAG or fine-tuning before you've tried a well-written prompt; it's the cheapest lever and often solves 80% of "the model isn't doing what I want".

When to combine both

Many production systems use both: fine-tune the model on a smaller set of examples so it always replies in your exact tone/format/domain jargon, AND use RAG so it always answers from fresh, citable facts. Fine-tuning handles the how; RAG handles the what.

The one-line interview answer: "Start with RAG; fine-tune only when prompting + RAG can't get you the behaviour you need — and even then the two are often combined, not either/or."

What we're building: A real AI Document Q&A Bot — upload any PDF, ask questions about it in plain English, get grounded answers with the exact source snippets shown underneath. This is the project every RAG interview question in this chapter (chunking, embeddings, retrieval, grounding, citations) comes together in. Tech stack: Streamlit (UI) + pypdf (text extraction) + ChromaDB (vector store, from Ch3) + OpenAI (embeddings + chat).

---

Step 0 — The pipeline, end to end (read first)

User uploads PDF
   -> pypdf extracts raw text
   -> text splitter chunks it (size 500, overlap 100)
   -> OpenAI embeds every chunk
   -> ChromaDB stores the vectors
   -> user types a question
   -> question gets embedded
   -> ChromaDB returns top-5 nearest chunks
   -> chunks + question go into a grounding prompt
   -> LLM answers ONLY from that context
   -> answer + source snippets shown to the user

Every step below is one link in that chain — notice what each step hands to the next.

---

Step 1 — Extract text from the uploaded PDF

from pypdf import PdfReader

def load_pdf(file):
    reader = PdfReader(file)
    text = ""
    for page in reader.pages:
        text += page.extract_text() + "\n"
    return text

What's happening: PdfReader(file) opens the uploaded PDF (Streamlit hands you a file-like object directly — no need to save it to disk first). reader.pages is a list of page objects; .extract_text() pulls the plain text out of each one. We glue every page into one long text string, with a \n between pages so page boundaries don't merge two words together. Hands to Step 2: one big string of raw document text.

🌍 Real-world example: this is exactly what a policy-document search tool at a company does with the HR handbook, or what a legal-tech tool does with a contract — turn a PDF nobody wants to read into searchable text.

---

Step 2 — Chunk the text (size 500, overlap 100)

from langchain.text_splitter import RecursiveCharacterTextSplitter

def create_chunks(text):
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=500, chunk_overlap=100
    )
    return splitter.split_text(text)

What's happening: one embedding for the whole PDF would be too vague to answer a specific question, so we split into ~500-character pieces. chunk_overlap=100 means each chunk repeats the last 100 characters of the previous one — so a sentence that gets cut in half at a chunk boundary is still whole in at least one chunk. Hands to Step 3: a Python list of chunk strings, e.g. ["chunk 0 text...", "chunk 1 text...", ...].

💡 Chunk overlap = the safety margin that stops a fact from being silently lost right at a chunk boundary.

---

Step 3 — Embed and store in ChromaDB

import chromadb

chroma = chromadb.PersistentClient(path="./vectordb")

def store_documents(chunks, collection_name="docs"):
    collection = chroma.get_or_create_collection(collection_name)
    collection.add(
        documents=chunks,
        ids=[f"chunk_{i}" for i in range(len(chunks))]
    )
    return collection

What's happening (Ch3 recap, one line): you already know ChromaDB turns text into embeddings and stores the vectors for similarity search. PersistentClient(path=...) saves the DB to disk so it survives between runs (not just in memory). collection.add(documents=chunks, ...) — pass it plain text and Chroma's default embedding function embeds each chunk for you and stores the vector + the original text + an id. Hands to Step 4: a searchable collection sitting on disk, ready for queries.

---

Step 4 — Retrieve top-5 chunks for a question

def ask_question(question, collection_name="docs"):
    collection = chroma.get_collection(collection_name)
    results = collection.query(query_texts=[question], n_results=5)
    context = "\n---\n".join(results["documents"][0])

What's happening: collection.query(query_texts=[question], n_results=5) embeds the question the same way the chunks were embedded, then returns the 5 chunks whose vectors are closest (cosine similarity, from Ch3). results["documents"][0] is a list of those 5 chunk strings; joining them with "\n---\n" builds one context block the LLM can read. Hands to Step 5: a context string containing the 5 most relevant slices of the PDF.

---

Step 5 — The grounding prompt (the anti-hallucination core)

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": f"""Answer based ONLY on this context.
If not found, say "Not in the documents."

Context:
{context}"""},
        {"role": "user", "content": question}
    ],
    temperature=0
)

What's happening: this is the single most important line in the whole project. The system message hard-instructs the model: use ONLY the context we retrieved, and admit when the answer isn't there — this is what stops the LLM from confidently making things up (hallucinating) from its general training knowledge instead of your document. temperature=0 makes the answer as deterministic/factual as possible, not creative. Hands to Step 6: a response object whose .choices[0].message.content is the grounded answer text.

🌍 Real-world example: this exact pattern — retrieve, then instruct "only from context" — is how every real internal-docs chatbot (Notion AI, a company's HR bot, a legal contract assistant) avoids inventing policy that doesn't exist.

---

Step 6 — Return the answer WITH source snippets

    return {
        "answer": response.choices[0].message.content,
        "sources": results["documents"][0][:3]
    }

What's happening: we don't just return the answer text — we also return the top 3 raw chunks that were used to generate it. This is what turns a black-box answer into a trustworthy, citable one: the user can expand "Sources" and see with their own eyes which lines of the PDF the bot is basing its answer on. Hands to Step 7: a dict {answer, sources} the frontend renders directly.

💡 Source citation = showing the exact retrieved text a claim came from, so the user (or an interviewer) can verify the answer isn't hallucinated.

---

Step 7 — Wire it to a Streamlit UI

import streamlit as st
from rag_engine import load_pdf, create_chunks, store_documents, ask_question

st.title("AI Document Q&A Bot")

uploaded_file = st.file_uploader("Upload PDF", type="pdf")
if uploaded_file:
    with st.spinner("Processing..."):
        text = load_pdf(uploaded_file)
        chunks = create_chunks(text)
        store_documents(chunks)
    st.success(f"Processed! {len(chunks)} chunks created.")

question = st.text_input("Ask a question about your document:")
if question:
    result = ask_question(question)
    st.write("### Answer:")
    st.write(result["answer"])
    with st.expander("Sources"):
        for i, source in enumerate(result["sources"]):
            st.write(f"**Source {i+1}:** {source[:200]}...")

What's happening — the full wiring, traced: st.file_uploader gives you the file object Step 1 consumes. Upload triggers Steps 1-3 (load_pdf -> create_chunks -> store_documents) inside a spinner, so ChromaDB is populated once, up front. st.text_input captures the question; every question triggers Steps 4-6 (ask_question = retrieve + grounding prompt + sources) fresh. The answer and expandable source list are the last link — turning a backend pipeline into something a non-technical user can actually use and trust.

---

Bonus features (resume/recruiter differentiators)

The base bot proves you can wire RAG end-to-end. These four separate you from every other "I built a RAG chatbot" resume line:

• Multi-file upload — accept several PDFs, tag each chunk's metadata with its source filename, so citations say which document an answer came from, not just which chunk.
• Chat history — store previous Q&A turns and pass recent turns into the prompt, handling follow-ups like "and page 3?" (this is conversational-rag, the next topic — query rewriting into a standalone question).
• Cohere re-ranking — retrieve wider (e.g. top 20) then re-rank with a cross-encoder down to the best 5 before building the prompt, for materially better answer quality on tricky questions.
• Token usage tracking — log response.usage per call so you can show real cost/latency numbers — exactly what a production team cares about, and a great interview talking point ("I optimized my RAG pipeline's token spend by X%").

What a recruiter sees in this project

• RAG implementation — the single hottest applied-AI skill in 2025-26 job postings.
• A real document processing pipeline — extraction, chunking, embedding, storage, all wired together.
• Vector database usage — ChromaDB in production-shaped code, not a toy example.
• Grounding + source attribution — proof you understand hallucination risk and how to mitigate it, which is exactly what separates a junior "I called an API" project from a senior-shaped one.
• A working, demoable UI — recruiters remember projects they can click through in 30 seconds; a Streamlit link in your resume is worth more than a GitHub repo they won't clone.

Standard definition: The AI Document Q&A Bot is a retrieval-augmented generation application that extracts text from an uploaded PDF, splits it into overlapping chunks, embeds and stores the chunks in a vector database, retrieves the most relevant chunks for a user's question, and generates an answer using a grounding prompt that restricts the LLM to the retrieved context — returning both the answer and the source snippets it was built from.

import chromadb
from openai import OpenAI
from pypdf import PdfReader
from langchain.text_splitter import RecursiveCharacterTextSplitter

client = OpenAI()
chroma = chromadb.PersistentClient(path="./vectordb")

def load_pdf(file):
    reader = PdfReader(file)
    text = ""
    for page in reader.pages:
        text += page.extract_text() + "\n"
    return text

def create_chunks(text):
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=500, chunk_overlap=100
    )
    return splitter.split_text(text)

def store_documents(chunks, collection_name="docs"):
    collection = chroma.get_or_create_collection(collection_name)
    collection.add(
        documents=chunks,
        ids=[f"chunk_{i}" for i in range(len(chunks))]
    )
    return collection

def ask_question(question, collection_name="docs"):
    collection = chroma.get_collection(collection_name)
    results = collection.query(query_texts=[question], n_results=5)

    context = "\n---\n".join(results["documents"][0])

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": f"""Answer based ONLY on this context.
If not found, say "Not in the documents."

Context:
{context}"""},
            {"role": "user", "content": question}
        ],
        temperature=0
    )

    return {
        "answer": response.choices[0].message.content,
        "sources": results["documents"][0][:3]
    }