← Back to DevBytes
AI July 18, 2026 13 min read

Building a Document Q&A Agent with LangChain: Complete Guide

Introduction: What is a Document Q&A Agent?

A Document Q&A Agent is an AI-powered system that can answer questions based on the content of your documents. Instead of relying on a language model's pre-trained knowledge (which may be outdated or hallucinated), the agent retrieves relevant chunks of text from your documents and uses them as context to generate accurate, grounded answers. LangChain provides the orchestration layer that connects document loading, text splitting, embedding generation, vector storage, retrieval, and response generation into a single cohesive pipeline.

At its core, the agent follows a pattern called Retrieval Augmented Generation (RAG): when a user asks a question, the system first retrieves the most relevant passages from the document store, then feeds those passages to a large language model (LLM) along with the original question to produce a well-informed answer.

Why Document Q&A Agents Matter

Document Q&A agents solve critical real-world problems that standalone LLMs cannot address:

Core Concepts and Architecture

Before diving into code, let's understand the key components of a LangChain Document Q&A system:

The data flow looks like this:

Documents → Load → Split → Embed → Store in Vector DB
                                      ↓
User Question → Embed → Similarity Search → Retrieved Chunks
                                      ↓
Question + Chunks → Prompt Template → LLM → Answer

Setting Up Your Environment

First, install the required packages. Create a new Python project and run:

pip install langchain langchain-openai langchain-community chromadb openai tiktoken pypdf unstructured

Set your OpenAI API key as an environment variable:

export OPENAI_API_KEY="your-api-key-here"

Or set it programmatically in your script (not recommended for production):

import os
os.environ["OPENAI_API_KEY"] = "your-api-key-here"

Step-by-Step Implementation

1. Loading Documents

LangChain supports dozens of document loaders. Here we'll demonstrate loading PDFs and text files—two of the most common formats.

Loading a PDF document:

from langchain_community.document_loaders import PyPDFLoader

# Load a PDF file - each page becomes a separate Document object
loader = PyPDFLoader("data/company_handbook.pdf")
documents = loader.load()

print(f"Loaded {len(documents)} pages")
print(f"First page preview: {documents[0].page_content[:200]}...")

Loading multiple PDFs from a directory:

from langchain_community.document_loaders import PyPDFDirectoryLoader

# Load all PDFs in a directory
loader = PyPDFDirectoryLoader("data/pdfs/")
documents = loader.load()
print(f"Loaded {len(documents)} pages from all PDFs")

Loading text files:

from langchain_community.document_loaders import TextLoader

loader = TextLoader("data/notes.txt")
documents = loader.load()

Loading web pages (bonus):

from langchain_community.document_loaders import WebBaseLoader

loader = WebBaseLoader("https://example.com/documentation")
documents = loader.load()

2. Splitting Documents into Chunks

Raw documents are often too large for embedding models and LLMs. We split them into overlapping chunks to preserve context at boundaries while keeping each chunk semantically coherent.

from langchain.text_splitter import RecursiveCharacterTextSplitter

# Create a text splitter with configurable chunk size and overlap
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,      # Maximum characters per chunk
    chunk_overlap=200,    # Overlap between consecutive chunks
    length_function=len,
    separators=["\n\n", "\n", ". ", " ", ""]  # Split on paragraphs first, then sentences
)

# Split the loaded documents
chunks = text_splitter.split_documents(documents)
print(f"Split {len(documents)} documents into {len(chunks)} chunks")

# Inspect a chunk
print(f"Chunk 0 content ({len(chunks[0].page_content)} chars):")
print(chunks[0].page_content[:300])
print(f"Metadata: {chunks[0].metadata}")

Choosing chunk size and overlap:

3. Creating Embeddings and Vector Stores

Now we embed each chunk and store the embeddings in a vector database. We'll use OpenAI's embedding model and ChromaDB (an open-source, in-memory vector store perfect for development).

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

# Initialize the embedding model
embeddings = OpenAIEmbeddings(
    model="text-embedding-3-small",  # Cost-effective, 1536 dimensions
    # For higher quality: "text-embedding-3-large" (3072 dimensions)
)

# Create a persistent Chroma vector store from the document chunks
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db"  # Persists to disk for reuse
)

print(f"Vector store created with {vectorstore._collection.count()} embeddings")

Loading an existing persisted vector store:

# On subsequent runs, load from disk instead of re-embedding
vectorstore = Chroma(
    persist_directory="./chroma_db",
    embedding_function=embeddings
)

Testing similarity search manually:

# Perform a raw similarity search
results = vectorstore.similarity_search(
    "What is the company's remote work policy?",
    k=4  # Number of chunks to retrieve
)

for i, doc in enumerate(results):
    print(f"\n--- Result {i+1} (Source: {doc.metadata.get('source', 'unknown')}) ---")
    print(doc.page_content[:200])

4. Building the Retrieval Chain

The retriever wraps the vector store and provides a clean interface. We then build a chain that takes a question, retrieves context, formats a prompt, and calls the LLM.

from langchain_openai import ChatOpenAI
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.prompts import ChatPromptTemplate

# Convert vectorstore to a retriever
retriever = vectorstore.as_retriever(
    search_type="similarity",  # Options: "similarity", "mmr", "similarity_score_threshold"
    search_kwargs={"k": 4}     # Retrieve top 4 chunks
)

# Initialize the LLM
llm = ChatOpenAI(
    model="gpt-4o",
    temperature=0.1  # Low temperature for factual accuracy
)

# Define the system prompt template
system_prompt = """You are a helpful assistant that answers questions based on the provided context.
Use only the information from the context to answer the question.
If the context doesn't contain the answer, say "I don't have enough information to answer that."
Always cite the source document and page when possible.

Context:
{context}"""

prompt = ChatPromptTemplate.from_messages([
    ("system", system_prompt),
    ("human", "{input}")
])

# Create the document combination chain
combine_docs_chain = create_stuff_documents_chain(
    llm=llm,
    prompt=prompt
)

# Create the full retrieval chain
rag_chain = create_retrieval_chain(
    retriever=retriever,
    combine_docs_chain=combine_docs_chain
)

print("RAG chain ready!")

Querying the chain:

# Ask a question
response = rag_chain.invoke({
    "input": "What benefits does the company offer for remote employees?"
})

print(f"Answer: {response['answer']}")
print(f"\nSources consulted:")
for i, doc in enumerate(response['context']):
    source = doc.metadata.get('source', 'unknown')
    page = doc.metadata.get('page', 'N/A')
    print(f"  [{i+1}] {source} (page {page})")

5. Adding Conversational Memory

To handle follow-up questions and maintain conversation context, we add memory that tracks the entire chat history.

from langchain.chains import create_history_aware_retriever
from langchain_core.prompts import MessagesPlaceholder
from langchain_core.messages import HumanMessage, AIMessage

# Prompt to rephrase the question using chat history for better retrieval
contextualize_q_prompt = ChatPromptTemplate.from_messages([
    ("system", """Given the chat history and the latest user question,
rephrase the question as a standalone query that captures all relevant context.
Do NOT answer the question, just rephrase it.
Standalone query:"""),
    MessagesPlaceholder("chat_history"),
    ("human", "{input}")
])

# Create a history-aware retriever
history_aware_retriever = create_history_aware_retriever(
    llm=llm,
    retriever=retriever,
    prompt=contextualize_q_prompt
)

# Full Q&A prompt with context
qa_prompt = ChatPromptTemplate.from_messages([
    ("system", system_prompt),
    MessagesPlaceholder("chat_history"),
    ("human", "{input}")
])

# Rebuild the combine chain with the new prompt
combine_docs_chain_with_history = create_stuff_documents_chain(
    llm=llm,
    prompt=qa_prompt
)

# Final conversational RAG chain
conversational_rag_chain = create_retrieval_chain(
    retriever=history_aware_retriever,
    combine_docs_chain=combine_docs_chain_with_history
)

6. Putting It All Together: The Complete Agent

Here's the complete, ready-to-run implementation that loads documents, builds the vector store, and provides an interactive Q&A interface with memory:

"""
Complete Document Q&A Agent with LangChain
Loads documents, builds a vector store, and runs an interactive Q&A loop with memory.
"""

import os
from langchain_community.document_loaders import PyPDFDirectoryLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Chroma
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain.chains import create_history_aware_retriever
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.messages import HumanMessage, AIMessage

# ── Configuration ──────────────────────────────────────────
DATA_DIR = "./documents"
CHROMA_PERSIST_DIR = "./chroma_db"
CHUNK_SIZE = 1000
CHUNK_OVERLAP = 200
RETRIEVAL_K = 4

# ── Step 1: Load documents ──────────────────────────────────
print("📄 Loading documents...")
loader = PyPDFDirectoryLoader(DATA_DIR)
documents = loader.load()
print(f"   Loaded {len(documents)} document pages")

# ── Step 2: Split into chunks ───────────────────────────────
print("✂️  Splitting into chunks...")
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=CHUNK_SIZE,
    chunk_overlap=CHUNK_OVERLAP,
    separators=["\n\n", "\n", ". ", " ", ""]
)
chunks = text_splitter.split_documents(documents)
print(f"   Created {len(chunks)} chunks")

# ── Step 3: Create embeddings and vector store ──────────────
print("🧠 Creating embeddings and vector store...")
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

# Only embed if vector store doesn't exist
if os.path.exists(CHROMA_PERSIST_DIR):
    print("   Loading existing vector store from disk...")
    vectorstore = Chroma(
        persist_directory=CHROMA_PERSIST_DIR,
        embedding_function=embeddings
    )
else:
    print("   Creating new vector store (this may take a while)...")
    vectorstore = Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
        persist_directory=CHROMA_PERSIST_DIR
    )
print(f"   Vector store ready with {vectorstore._collection.count()} embeddings")

# ── Step 4: Set up retriever and LLM ────────────────────────
retriever = vectorstore.as_retriever(
    search_type="similarity",
    search_kwargs={"k": RETRIEVAL_K}
)

llm = ChatOpenAI(model="gpt-4o", temperature=0.1)

# ── Step 5: Build prompts ───────────────────────────────────
contextualize_q_prompt = ChatPromptTemplate.from_messages([
    ("system", """Given the chat history and the latest user question,
rephrase the question into a standalone query that captures all context.
Do NOT answer, only rephrase. Standalone query:"""),
    MessagesPlaceholder("chat_history"),
    ("human", "{input}")
])

qa_system_prompt = """You are a helpful assistant answering questions based on provided context.
Use ONLY the context below. If the answer isn't in the context, say so honestly.
Cite source documents when possible.

Context:
{context}"""

qa_prompt = ChatPromptTemplate.from_messages([
    ("system", qa_system_prompt),
    MessagesPlaceholder("chat_history"),
    ("human", "{input}")
])

# ── Step 6: Build chains ────────────────────────────────────
history_aware_retriever = create_history_aware_retriever(
    llm=llm,
    retriever=retriever,
    prompt=contextualize_q_prompt
)

combine_docs_chain = create_stuff_documents_chain(
    llm=llm,
    prompt=qa_prompt
)

rag_chain = create_retrieval_chain(
    retriever=history_aware_retriever,
    combine_docs_chain=combine_docs_chain
)

# ── Step 7: Interactive Q&A loop ────────────────────────────
print("\n✅ Agent ready! Type your questions below.")
print("   Type 'exit' to quit, 'clear' to reset history.\n")

chat_history = []

while True:
    question = input("🙋 You: ")
    if question.lower() in ("exit", "quit"):
        break
    if question.lower() == "clear":
        chat_history = []
        print("   History cleared.\n")
        continue

    response = rag_chain.invoke({
        "input": question,
        "chat_history": chat_history
    })

    answer = response["answer"]
    sources = response["context"]

    print(f"\n🤖 Agent: {answer}")
    print(f"\n   📚 Sources:")
    seen = set()
    for doc in sources:
        source = doc.metadata.get("source", "unknown")
        page = doc.metadata.get("page", "N/A")
        key = f"{source}:{page}"
        if key not in seen:
            seen.add(key)
            print(f"      • {source} (page {page})")
    print()

    # Update history
    chat_history.append(HumanMessage(content=question))
    chat_history.append(AIMessage(content=answer))

Advanced Techniques

Handling Multiple Document Types

Real-world projects often involve mixed document formats. LangChain's document loaders handle this gracefully:

from langchain_community.document_loaders import (
    PyPDFLoader,
    TextLoader,
    CSVLoader,
    UnstructuredMarkdownLoader,
    UnstructuredHTMLLoader
)

all_documents = []

# Load PDFs
pdf_loader = PyPDFDirectoryLoader("data/pdfs/")
all_documents.extend(pdf_loader.load())

# Load text files
import glob
for txt_file in glob.glob("data/*.txt"):
    loader = TextLoader(txt_file)
    all_documents.extend(loader.load())

# Load CSV (each row becomes a document)
csv_loader = CSVLoader("data/products.csv")
all_documents.extend(csv_loader.load())

# Load Markdown
md_loader = UnstructuredMarkdownLoader("data/README.md")
all_documents.extend(md_loader.load())

print(f"Total documents loaded: {len(all_documents)}")
# Now split and embed all_documents as before

Using Different Retrieval Strategies

The basic similarity search works well, but LangChain offers alternative retrieval methods that can improve results:

# Strategy 1: Maximum Marginal Relevance (MMR) - reduces redundancy
mmr_retriever = vectorstore.as_retriever(
    search_type="mmr",
    search_kwargs={
        "k": 6,
        "fetch_k": 20,       # Fetch more candidates then filter
        "lambda_mult": 0.7   # 0 = max diversity, 1 = max similarity
    }
)

# Strategy 2: Similarity score threshold - only return highly relevant chunks
threshold_retriever = vectorstore.as_retriever(
    search_type="similarity_score_threshold",
    search_kwargs={
        "k": 10,
        "score_threshold": 0.5  # Only return chunks with similarity >= 0.5
    }
)

# Strategy 3: Ensemble retriever - combine multiple retrievers
from langchain.retrievers import EnsembleRetriever

keyword_retriever = ...  # BM25 or other keyword-based retriever
ensemble = EnsembleRetriever(
    retrievers=[mmr_retriever, keyword_retriever],
    weights=[0.7, 0.3]  # Weighted combination
)

Optimizing with Reranking

For production-quality retrieval, add a reranking step: retrieve a larger pool of candidates, then use a more powerful (but slower) model to rerank and select the best ones:

from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import CrossEncoderReranker
from langchain_community.cross_encoders import HuggingFaceCrossEncoder

# Initialize a cross-encoder reranker
model = HuggingFaceCrossEncoder(model_name="BAAI/bge-reranker-base")
compressor = CrossEncoderReranker(model=model, top_n=4)

# Wrap the base retriever with reranking
base_retriever = vectorstore.as_retriever(search_kwargs={"k": 20})  # Fetch more
compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=base_retriever
)

# Use compression_retriever in your chain instead of the raw retriever
# The retriever now: fetches 20 candidates → reranks them → returns top 4

Building a Full Agent with Tools (Advanced Pattern)

For complex scenarios where the agent needs to decide how to answer (search documents, summarize, compare), use the Agent pattern instead of a fixed chain:

from langchain.agents import create_openai_tools_agent, AgentExecutor
from langchain.tools import tool
from langchain_core.prompts import ChatPromptTemplate

@tool
def search_documents(query: str) -> str:
    """Search the company knowledge base for information relevant to the query."""
    docs = retriever.invoke(query)
    return "\n\n---\n\n".join([d.page_content for d in docs[:3]])

@tool
def list_document_sources() -> str:
    """Return a list of all document sources available in the knowledge base."""
    # Get unique sources from the vector store
    sources = set()
    for doc_id in vectorstore._collection.get()["metadatas"]:
        sources.add(doc_id.get("source", "unknown"))
    return "\n".join(sorted(sources))

tools = [search_documents, list_document_sources]

agent_prompt = ChatPromptTemplate.from_messages([
    ("system", """You are a helpful document Q&A agent.
Use the search_documents tool to find relevant information before answering.
Always cite sources. If multiple documents are relevant, compare them."""),
    MessagesPlaceholder("chat_history"),
    ("human", "{input}"),
    ("placeholder", "{agent_scratchpad}")
])

agent = create_openai_tools_agent(llm, tools, agent_prompt)
agent_executor = AgentExecutor(
    agent=agent,
    tools=tools,
    verbose=True,
    max_iterations=5
)

# Query the agent
result = agent_executor.invoke({
    "input": "Compare the remote work policies from 2023 and 2024. What changed?",
    "chat_history": []
})
print(result["output"])

Best Practices

Conclusion

Building a Document Q&A Agent with LangChain transforms static documents into interactive knowledge bases. The RAG architecture—loading, splitting, embedding, storing, retrieving, and generating—provides a powerful pattern that combines the strengths of vector search with the reasoning capabilities of large language models. By following this guide, you've built a complete agent capable of ingesting PDFs and other documents, handling conversational context with memory, and delivering grounded, source-cited answers. The modular design of LangChain lets you swap components easily: change the embedding model, switch vector stores from Chroma to Pinecone for production scale, upgrade the LLM, or add reranking for higher precision. Start with the simple pipeline demonstrated here, iterate on your chunk sizes and prompts based on real user feedback, and gradually adopt advanced techniques like agent-based tool use and cross-encoder reranking as your needs grow. The combination of LangChain's orchestration with modern embedding and LLM APIs gives you a production-ready foundation for building intelligent document assistants that truly understand your content.

— Ad —

Google AdSense will appear here after approval

← Back to all articles