← Back to DevBytes
AI July 16, 2026 12 min read

Migrating from Legacy Frameworks to LlamaIndex

Migrating from Legacy Frameworks to LlamaIndex: A Complete Developer Tutorial

When your AI application outgrows its original scaffolding, migrating from a legacy LLM framework to LlamaIndex can unlock significant performance gains, cleaner architecture, and a more intuitive developer experience. This tutorial walks you through a complete migration—covering everything from initial audit to final deployment—with practical code examples you can adapt to your own codebase.

What Is LlamaIndex and Why Migrate?

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

LlamaIndex is a data framework designed specifically for building LLM-powered applications that connect large language models to external data sources. Unlike general-purpose orchestration frameworks, LlamaIndex places data indexing and retrieval at the center of its architecture. It provides a rich set of components—from document loaders and node parsers to sophisticated query engines and agents—all built around a unified data model.

Legacy frameworks such as LangChain, custom RAG pipelines built with raw vector libraries, or first-generation chatbot toolkits often suffer from several pain points that LlamaIndex directly addresses:

Understanding the LlamaIndex Advantage

LlamaIndex introduces a clean separation of concerns through its core abstractions. Every piece of data flows through a consistent pipeline: loading → parsing → indexing → querying. This design makes it straightforward to swap components, experiment with different chunking strategies, or upgrade your embedding model without rewriting retrieval logic. Additionally, LlamaIndex offers built-in observability, structured data extraction, and a powerful ingestion pipeline that handles incremental updates natively—features that typically require extensive custom code in legacy setups.

Migration Path: From Legacy Frameworks to LlamaIndex

A successful migration follows a structured, step-by-step approach. The process below assumes you are migrating from a LangChain-based RAG application (the most common legacy scenario), but the principles apply equally to custom-built pipelines using tools like ChromaDB, Pinecone with manual embedding logic, or older chatbot frameworks.

Step 1: Auditing Your Current Implementation

Before writing any migration code, catalog every component in your existing system. Create an inventory that maps each legacy abstraction to its LlamaIndex counterpart. A typical audit table looks like this:

Here is a concrete example of a legacy LangChain RAG pipeline that we will migrate step by step:

# Legacy LangChain RAG implementation
from langchain.document_loaders import DirectoryLoader, TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.chains import RetrievalQA
from langchain.chat_models import ChatOpenAI

# Load documents
loader = DirectoryLoader('./docs/', glob="**/*.txt", loader_cls=TextLoader)
documents = loader.load()

# Split documents
text_splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=50)
chunks = text_splitter.split_documents(documents)

# Create vector store
embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(chunks, embeddings, persist_directory="./chroma_db")

# Build QA chain
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})
llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=retriever,
    chain_type="stuff",
    return_source_documents=True
)

# Query
result = qa_chain.invoke({"query": "What is the refund policy?"})
print(result['result'])

This pipeline works, but it tightly couples loading, chunking, embedding, and retrieval into a single procedural block. Adding features like metadata filtering, hybrid search, or multi-document synthesis requires significant refactoring. Now let's migrate it to LlamaIndex.

Step 2: Mapping Legacy Abstractions to LlamaIndex

LlamaIndex uses a ServiceContext to encapsulate global configuration—embedding model, LLM, chunk size, and other defaults. This replaces the scattered configuration objects in LangChain. The ServiceContext is then passed to index constructors and query engines, ensuring consistency across your entire pipeline.

# LlamaIndex: Centralized configuration with ServiceContext
from llama_index.core import ServiceContext, Settings
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI

# Option 1: Global settings (recommended for simpler apps)
Settings.llm = OpenAI(model="gpt-3.5-turbo", temperature=0)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-ada-002")
Settings.chunk_size = 512
Settings.chunk_overlap = 50

# Option 2: Explicit ServiceContext (for multiple configurations)
service_context = ServiceContext.from_defaults(
    llm=OpenAI(model="gpt-3.5-turbo", temperature=0),
    embed_model=OpenAIEmbedding(),
    chunk_size=512,
    chunk_overlap=50,
)
# You can also add custom node parsers, transformations, etc.
# service_context = ServiceContext.from_defaults(
#     llm=my_llm,
#     embed_model=my_embed_model,
#     node_parser=SentenceSplitter(chunk_size=256),
# )

Step 3: Replacing Document Loaders with LlamaIndex Readers

LlamaIndex provides a rich ecosystem of data connectors (readers) that handle everything from local files to cloud storage, APIs, and databases. The SimpleDirectoryReader is the most versatile starting point—it automatically detects file types and selects the appropriate parser. For advanced use cases, you can chain multiple readers or create custom ones.

# LlamaIndex document loading
from llama_index.core import SimpleDirectoryReader

# Simple: Load all supported files from a directory
documents = SimpleDirectoryReader(
    input_dir="./docs/",
    recursive=True,
    required_exts=[".txt", ".pdf", ".md", ".csv"]  # optional filtering
).load_data()

print(f"Loaded {len(documents)} documents")

# For specific file types, use specialized readers
from llama_index.readers.file import PDFReader

pdf_reader = PDFReader()
pdf_documents = pdf_reader.load_data(file_path="./docs/policies.pdf")

# You can also load from Notion, Google Drive, Slack, and more
# from llama_index.readers.notion import NotionPageReader
# reader = NotionPageReader(integration_token="...")
# notion_docs = reader.load_data(page_ids=["page_id_1"])

Step 4: Migrating Indexing and Retrieval Logic

This is where LlamaIndex truly shines. Instead of manually splitting documents and inserting them into a vector store, you create an index—a high-level abstraction that handles chunking, embedding, and storage in one call. LlamaIndex supports numerous index types: VectorStoreIndex (dense retrieval), SummaryIndex (full-document summarization), TreeIndex (hierarchical traversal), and KeywordTableIndex (BM25-style sparse retrieval). For most RAG use cases, VectorStoreIndex is the direct replacement.

# LlamaIndex: Creating an index from documents
from llama_index.core import VectorStoreIndex
from llama_index.vector_stores.chroma import ChromaVectorStore
import chromadb

# Option A: In-memory index (great for prototyping)
index = VectorStoreIndex.from_documents(documents)

# Option B: Persistent index with Chroma (matching our legacy setup)
chroma_client = chromadb.PersistentClient(path="./chroma_db_llamaindex")
chroma_collection = chroma_client.get_or_create_collection("docs_collection")
vector_store = ChromaVectorStore(
    chroma_collection=chroma_collection,
    # Optional: custom metadata filtering setup
)
index = VectorStoreIndex.from_vector_store(
    vector_store=vector_store,
    # If you need to re-index from documents:
    # documents=documents,
)

# Persist the index metadata to disk for faster reloading
index.storage_context.persist(persist_dir="./llama_index_storage")
print(f"Index created with {index.summary}")

Now let's replace the retrieval component. In the legacy code, we called vectorstore.as_retriever() and passed it to a chain. LlamaIndex elevates retrieval to a first-class query engine concept that handles retrieval, reranking, and response synthesis in a single coherent interface.

# LlamaIndex: Query engine replaces RetrievalQA chain
query_engine = index.as_query_engine(
    similarity_top_k=4,           # Equivalent to search_kwargs={"k": 4}
    response_mode="compact",      # Options: "tree_summarize", "refine", "simple_summarize"
    verbose=True,                 # See retrieved nodes in logs
)

# Simple query
response = query_engine.query("What is the refund policy?")
print(response.response)  # The synthesized answer

# Access source nodes (equivalent to source_documents)
for source_node in response.source_nodes:
    print(f"Source: {source_node.node.metadata.get('file_name')}")
    print(f"Score: {source_node.score}")
    print(f"Text preview: {source_node.node.text[:200]}...")

Step 5: Advanced Retrieval – Adding Reranking and Hybrid Search

One of the strongest reasons to migrate is LlamaIndex's built-in support for advanced retrieval techniques that require significant custom code in legacy frameworks. Let's add a cross-encoder reranker and hybrid (dense + sparse) search—features that dramatically improve retrieval quality for production applications.

# Advanced retrieval with reranking and hybrid search
from llama_index.core.postprocessor import SentenceTransformerRerank
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core import get_response_synthesizer

# 1. Create a retriever with hybrid search (dense + BM25 sparse)
from llama_index.retrievers.bm25 import BM25Retriever
from llama_index.core.retrievers import QueryFusionRetriever

# Build BM25 retriever on the same nodes
bm25_retriever = BM25Retriever.from_defaults(
    nodes=index.docstore.get_all_nodes(),
    similarity_top_k=8,
)

# Dense retriever from vector index
vector_retriever = VectorIndexRetriever(
    index=index,
    similarity_top_k=8,
)

# Combine with fusion (reciprocal rank fusion)
fusion_retriever = QueryFusionRetriever(
    retrievers=[vector_retriever, bm25_retriever],
    similarity_top_k=10,
    num_queries=1,
    mode="reciprocal_rerank",  # Or "relative_score", "dist_based_score"
)

# 2. Add a cross-encoder reranker for final precision
reranker = SentenceTransformerRerank(
    model="cross-encoder/ms-marco-MiniLM-L-6-v2",
    top_n=4,
)

# 3. Configure response synthesis
response_synthesizer = get_response_synthesizer(
    response_mode="compact",
    verbose=True,
)

# 4. Assemble the query engine
advanced_query_engine = RetrieverQueryEngine(
    retriever=fusion_retriever,
    response_synthesizer=response_synthesizer,
    node_postprocessors=[reranker],
)

# Query with the advanced pipeline
response = advanced_query_engine.query(
    "What are the conditions for a full refund?"
)
print(response.response)

This entire advanced pipeline—hybrid retrieval, fusion scoring, cross-encoder reranking, and response synthesis—is configured declaratively. In a legacy framework, each of these steps would require separate classes, manual score aggregation, and custom post-processing logic spread across multiple files.

Step 6: Rebuilding Chat and Agent Logic

Legacy conversational RAG implementations typically involve complex chains with conversation memory, condensation prompts, and tool-calling agents. LlamaIndex simplifies this with dedicated chat engines and agent classes that handle conversation history, follow-up questions, and multi-turn reasoning out of the box.

# Migrating from ConversationalRetrievalChain to LlamaIndex ChatEngine
from llama_index.core.memory import ChatMemoryBuffer
from llama_index.core.chat_engine import CondensePlusContextChatEngine

# Create a chat memory buffer (keeps conversation context)
chat_memory = ChatMemoryBuffer.from_defaults(
    token_limit=3000,  # Automatically prunes older messages
)

# Build a chat engine that condenses follow-up questions
chat_engine = CondensePlusContextChatEngine.from_defaults(
    retriever=index.as_retriever(similarity_top_k=5),
    memory=chat_memory,
    llm=Settings.llm,
    verbose=True,
)

# Multi-turn conversation
response1 = chat_engine.chat("What is your return policy?")
print("Assistant:", response1.response)

response2 = chat_engine.chat("Does that apply to electronics as well?")
# The engine automatically condenses this to:
# "Does the return policy apply to electronics?"
# then retrieves relevant context and synthesizes a coherent answer
print("Assistant:", response2.response)

# Reset conversation when needed
chat_engine.reset()

For agent-based workflows (tool calling, multi-step reasoning), LlamaIndex offers native agent classes that integrate seamlessly with your indexed data and external tools:

# Migrating from LangChain agents to LlamaIndex agents
from llama_index.core.agent import ReActAgent
from llama_index.core.tools import QueryEngineTool, FunctionTool

# Convert your query engine into a tool the agent can use
query_tool = QueryEngineTool.from_defaults(
    query_engine=advanced_query_engine,
    name="document_qa",
    description="Searches company documents to answer policy questions",
)

# Add custom tools (replaces legacy Tool classes)
def calculate_discount(original_price: float, discount_percentage: float) -> float:
    """Calculate discounted price given original price and discount percentage."""
    return original_price * (1 - discount_percentage / 100)

discount_tool = FunctionTool.from_defaults(
    fn=calculate_discount,
    name="discount_calculator",
    description="Calculates the final price after applying a discount percentage",
)

# Create the ReAct agent
agent = ReActAgent.from_tools(
    tools=[query_tool, discount_tool],
    llm=Settings.llm,
    verbose=True,
    max_iterations=10,  # Prevent infinite loops
)

# Agent autonomously decides which tools to call
response = agent.chat(
    "Find the refund policy for electronics. If a $200 item is eligible, "
    "calculate the refund after a 15% restocking fee."
)
print(response.response)

# Inspect the agent's reasoning steps
for task in agent.get_completed_tasks():
    print(f"Thought: {task.thought}")
    print(f"Action: {task.action}")
    print(f"Observation: {task.observation}")

Step 7: Handling Structured Data Extraction

Many legacy pipelines struggle with extracting structured data (JSON, tables, key-value pairs) from unstructured text. LlamaIndex provides dedicated extractors that combine retrieval with structured output parsing:

# Structured data extraction from documents
from llama_index.core.extractors import (
    MetadataExtractor,
    QuestionsAnsweredExtractor,
    SummaryExtractor,
)
from llama_index.core.node_parser import SentenceSplitter

# Define extractors to run during ingestion
metadata_extractor = MetadataExtractor(
    extractors=[
        QuestionsAnsweredExtractor(questions=3, llm=Settings.llm),
        SummaryExtractor(summaries=["prev", "self"], llm=Settings.llm),
    ],
)

# Re-index with metadata extraction
node_parser = SentenceSplitter(chunk_size=512, chunk_overlap=50)

index_with_metadata = VectorStoreIndex.from_documents(
    documents,
    transformations=[node_parser, metadata_extractor],
)

# Query with structured output
from llama_index.core.prompts import PromptTemplate
from pydantic import BaseModel, Field

class RefundPolicy(BaseModel):
    """Structured representation of a refund policy."""
    eligible_items: str = Field(description="Types of items eligible for refund")
    time_limit_days: int = Field(description="Number of days for refund eligibility")
    restocking_fee_percentage: float = Field(description="Restocking fee as percentage")
    exceptions: str = Field(description="Any exceptions to the policy")

query_engine = index_with_metadata.as_query_engine(
    output_cls=RefundPolicy,
    response_mode="tree_summarize",
)

response = query_engine.query("Extract the refund policy details")
print(response.response)  # Plain text answer
print(response.output)    # Parsed RefundPolicy object
# Output: RefundPolicy(
#   eligible_items='All items except clearance',
#   time_limit_days=30,
#   restocking_fee_percentage=15.0,
#   exceptions='Electronics require original packaging'
# )

Best Practices for a Smooth Migration

Based on real-world migration experiences, here are the practices that consistently lead to successful outcomes:

# Example: Evaluating migration quality quantitatively
from llama_index.core.evaluation import FaithfulnessEvaluator, RelevancyEvaluator
import json

# Load test queries (migrate these from your legacy test suite)
test_queries = [
    {"query": "What is the refund policy?", "reference": "Full refund within 30 days..."},
    {"query": "How to cancel subscription?", "reference": "Cancel via account settings..."},
    # ... more test cases
]

faithfulness_eval = FaithfulnessEvaluator(llm=Settings.llm)
relevancy_eval = RelevancyEvaluator(llm=Settings.llm)

results = []
for test_case in test_queries:
    response = advanced_query_engine.query(test_case["query"])
    
    faithfulness_result = faithfulness_eval.evaluate(
        response=response.response,
        contexts=[n.node.text for n in response.source_nodes],
    )
    relevancy_result = relevancy_eval.evaluate(
        response=response.response,
        contexts=[n.node.text for n in response.source_nodes],
        query=test_case["query"],
    )
    
    results.append({
        "query": test_case["query"],
        "faithfulness_passed": faithfulness_result.passing,
        "relevancy_passed": relevancy_result.passing,
        "response": response.response,
    })

print(json.dumps(results, indent=2))
# Use these metrics to confirm migration quality before decommissioning legacy system

Conclusion

Migrating from a legacy LLM framework to LlamaIndex is not merely a library swap—it's an opportunity to rethink your data pipeline with cleaner abstractions, more powerful retrieval strategies, and built-in support for features that previously required extensive custom code. The structured migration path outlined in this tutorial—auditing your existing implementation, mapping abstractions, replacing loaders and indexers, upgrading query engines, rebuilding chat and agent logic, and validating with quantitative evaluation—provides a reliable blueprint for a successful transition. By embracing LlamaIndex's data-centric architecture, you position your application to scale gracefully, deliver higher-quality responses through advanced retrieval techniques like hybrid search and reranking, and significantly reduce the maintenance burden of hand-rolled RAG pipelines. Start with a small pilot, measure rigorously, and let the framework's well-designed abstractions guide your application toward a more robust and maintainable future.

🚀 Need a reliable AI agent for your project?

Deploy Hermes Agent in 10 minutes. Managed hosting, zero DevOps.

Get Started — $23.99/mo
← Back to all articles