Introduction to LangChain and Customer Support Bots
A customer support bot powered by LangChain represents a significant leap beyond traditional rule-based chatbots. LangChain is an open-source framework that simplifies building applications with large language models (LLMs) by providing modular components for chaining together prompts, memory, tools, and data retrieval. When applied to customer support, LangChain enables bots that can understand context, access knowledge bases, maintain conversation history, and take meaningful actions — all while staying grounded in your company's actual support documentation.
Why LangChain Matters for Customer Support
Traditional support bots rely on rigid decision trees or keyword matching, which frustrates customers when their questions fall outside predefined paths. LangChain transforms this by enabling:
- Contextual understanding — The bot interprets nuanced questions rather than requiring exact keyword matches
- Knowledge base integration — LangChain connects directly to your documentation, FAQs, and support articles via retrievers and vector stores
- Conversational memory — The bot remembers previous exchanges within a session, avoiding repetitive questions
- Tool use — The bot can check order status, create tickets, or fetch account details through API integrations
- Chain composability — You can combine multiple LLM calls, data sources, and logic steps into a single coherent workflow
Setting Up Your Environment
Before building the bot, install the required dependencies. This tutorial uses Python, LangChain, OpenAI as the LLM provider, and ChromaDB as the vector store for knowledge retrieval.
# Create a virtual environment
python -m venv support_bot_env
source support_bot_env/bin/activate # On Windows: support_bot_env\Scripts\activate
# Install core dependencies
pip install langchain langchain-openai chromadb tiktoken
# Optional: install additional tools for production
pip install streamlit # for a quick web UI
pip install python-dotenv # for managing environment variables
Set up your OpenAI API key as an environment variable. Create a .env file:
OPENAI_API_KEY=your-openai-api-key-here
Load this in your Python script:
import os
from dotenv import load_dotenv
load_dotenv()
# LangChain automatically reads OPENAI_API_KEY from the environment
# but you can also set it explicitly:
# os.environ["OPENAI_API_KEY"] = "your-key-here"
Building the Core Components
1. Creating a Basic LLM Chain for Support Responses
Start with the simplest building block: a prompt template that structures the LLM's response for customer support scenarios. This chain takes a customer question and produces a helpful answer.
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
# Initialize the LLM with controlled temperature for consistent support responses
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.3)
# Craft a prompt template designed for customer support
prompt = ChatPromptTemplate.from_messages([
("system", """You are a helpful and empathetic customer support agent for TechGadget Inc.,
a company that sells consumer electronics. Your tone is friendly, professional, and patient.
When you don't know an answer, honestly admit it and offer to escalate the issue.
Never make up product specifications or policies. If a customer is frustrated, acknowledge
their feelings before providing solutions."""),
("human", "{customer_question}")
])
# Create a simple chain: prompt -> llm -> string output
basic_chain = prompt | llm | StrOutputParser()
# Test the chain
response = basic_chain.invoke({"customer_question": "My wireless earbuds won't connect to my phone. What should I do?"})
print(response)
This basic chain works for general questions, but it lacks access to your actual product information and can't handle multi-turn conversations. Let's fix both limitations.
2. Adding Conversational Memory
Customer support conversations rarely consist of a single question. Users often provide context over multiple messages, and the bot must track this history. LangChain provides several memory classes; for support bots, ConversationSummaryBufferMemory works well because it keeps a rolling summary of older messages while preserving recent exchanges in full.
from langchain.memory import ConversationSummaryBufferMemory
from langchain_core.runnables import RunnableLambda
from langchain_core.messages import HumanMessage, AIMessage
# Initialize memory with summary capabilities
memory = ConversationSummaryBufferMemory(
llm=llm,
max_token_limit=400, # keep the last ~400 tokens in full, summarize older messages
return_messages=True,
memory_key="chat_history"
)
# Function to retrieve and format history for the prompt
def get_history(_):
# memory.load_memory_variables returns a dict with the chat_history key
return memory.load_memory_variables({})["chat_history"]
# Updated prompt that includes conversation history
prompt_with_memory = ChatPromptTemplate.from_messages([
("system", """You are a helpful and empathetic customer support agent for TechGadget Inc.
Use the conversation history to avoid asking for information the customer already provided.
{chat_history}"""),
("human", "{customer_question}")
])
# Build the conversational chain
def build_conversational_chain():
# RunnableLambda wraps the get_history function so it works in a chain
history_lambda = RunnableLambda(get_history)
# Chain: extract history -> format prompt with history -> LLM -> parse output
chain = history_lambda | prompt_with_memory | llm | StrOutputParser()
return chain
conversational_chain = build_conversational_chain()
# Simulate a multi-turn conversation
def process_message(chain, user_input):
response = chain.invoke({"customer_question": user_input})
# Save both the user input and assistant response to memory
memory.save_context({"input": user_input}, {"output": response})
return response
# Example conversation
print(process_message(conversational_chain, "I bought a SmartWatch Pro last week. The screen flickers sometimes."))
print(process_message(conversational_chain, "Yes, I already tried restarting it. It still happens."))
print(process_message(conversational_chain, "What's my warranty coverage for this?"))
The bot now remembers that the customer owns a SmartWatch Pro, has already tried restarting, and is asking about warranty coverage — all without the customer repeating themselves.
3. Connecting to a Knowledge Base with Retrieval-Augmented Generation (RAG)
The most critical capability for a support bot is accessing your actual product documentation. RAG allows the bot to search through your support articles, product manuals, and FAQs, then incorporate that information into its responses. This keeps answers accurate and grounded.
First, prepare your knowledge base documents and create embeddings:
from langchain_openai import OpenAIEmbeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
from langchain_core.documents import Document
# Sample support documents (in production, load these from files or a database)
support_docs = [
Document(
page_content="""SmartWatch Pro Troubleshooting Guide:
Screen Flickering Issue: If your SmartWatch Pro screen flickers, first perform a soft reset
by holding the side button for 10 seconds. If flickering persists, check for software updates
in the companion app under Settings > Device > Check for Updates. A known issue in firmware
v2.1 caused flickering on certain units — update to v2.2 resolves this. If problems continue
after updating, contact support for a replacement evaluation within warranty period.""",
metadata={"source": "troubleshooting-guide", "product": "SmartWatch Pro"}
),
Document(
page_content="""Warranty Policy: TechGadget Inc. provides a 1-year limited warranty
on all SmartWatch models from the date of purchase. The warranty covers manufacturing defects
including screen malfunctions, battery issues, and sensor failures. Accidental damage, water
damage beyond rated specifications, and unauthorized repairs void the warranty. Customers must
provide proof of purchase for warranty claims. Replacement units are shipped within 3-5 business
days after claim approval.""",
metadata={"source": "warranty-policy", "product": "all"}
),
Document(
page_content="""Wireless Earbuds X2 Connection Guide:
To pair your Wireless Earbuds X2: 1) Place both earbuds in the charging case and close the lid.
2) Open the lid and press the pairing button on the case for 3 seconds until the LED flashes white.
3) On your phone, go to Bluetooth settings and select 'TechGadget-X2' from the list.
If connection fails, reset the earbuds by pressing the button for 10 seconds until the LED flashes
red, then try pairing again. Ensure your phone's Bluetooth is version 4.0 or higher.""",
metadata={"source": "earbuds-guide", "product": "Wireless Earbuds X2"}
),
Document(
page_content="""Return and Exchange Policy: Customers may return products within 30 days
of delivery for a full refund. Products must be in original packaging with all accessories.
A 15% restocking fee applies to opened electronics unless the return is due to a defect.
Exchanges for defective products are processed free of charge. Return shipping is free for
defective items; customers pay return shipping for change-of-mind returns.""",
metadata={"source": "return-policy", "product": "all"}
),
]
# Split documents into smaller chunks for better retrieval
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=500,
chunk_overlap=50,
separators=["\n\n", "\n", ". ", " ", ""]
)
split_docs = text_splitter.split_documents(support_docs)
# Create embeddings and store in Chroma vector database
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(
documents=split_docs,
embedding=embeddings,
collection_name="support_knowledge_base",
persist_directory="./support_db" # persists to disk for reuse
)
# Create a retriever that fetches relevant documents
retriever = vectorstore.as_retriever(
search_type="similarity",
search_kwargs={"k": 3} # retrieve top 3 most relevant chunks
)
Now integrate the retriever into your chain. LangChain provides convenient factory functions for creating RAG chains:
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser
# Format retrieved documents into a single context string
def format_docs(docs):
return "\n\n".join(f"Source ({doc.metadata.get('source', 'unknown')}):\n{doc.page_content}" for doc in docs)
# RAG prompt that instructs the LLM to use retrieved context
rag_prompt = ChatPromptTemplate.from_messages([
("system", """You are a knowledgeable customer support agent for TechGadget Inc.
Use ONLY the following retrieved support documentation to answer the customer's question.
If the documentation doesn't contain the answer, say: "I don't have specific information
on that in our documentation. Let me escalate this to our specialist team."
Retrieved Documentation:
{context}
Conversation History:
{chat_history}"""),
("human", "{customer_question}")
])
# Build the full RAG chain with memory
def build_rag_chain():
# Step 1: Retrieve relevant documents based on the customer question
# Step 2: Format documents into context string
# Step 3: Get conversation history
# Step 4: Combine everything into the prompt
# Step 5: Generate response via LLM
# Step 6: Parse to string
rag_chain = (
{
"context": lambda x: format_docs(retriever.invoke(x["customer_question"])),
"chat_history": lambda x: get_history(x),
"customer_question": lambda x: x["customer_question"]
}
| rag_prompt
| llm
| StrOutputParser()
)
return rag_chain
rag_chain = build_rag_chain()
# Process a message with full RAG and memory
def process_rag_message(chain, user_input):
response = chain.invoke({"customer_question": user_input})
memory.save_context({"input": user_input}, {"output": response})
return response
# Test the RAG-powered bot
print(process_rag_message(rag_chain, "My SmartWatch Pro screen keeps flickering after restart. Help!"))
print(process_rag_message(rag_chain, "Am I still covered under warranty? I bought it 8 months ago."))
print(process_rag_message(rag_chain, "Great, how long will the replacement take to arrive?"))
This RAG chain now searches your actual support documentation, retrieves relevant information, and incorporates it into responses — while also maintaining conversation memory. The bot won't hallucinate product details because it's grounded in your documents.
4. Adding Tools for Action-Oriented Support
Beyond answering questions, a support bot should perform actions: look up order statuses, create support tickets, or check inventory. LangChain's tool-calling capabilities enable this. Define functions as tools that the LLM can choose to invoke.
from langchain_core.tools import tool
from typing import Optional
import json
# Define tools as Python functions with the @tool decorator
@tool
def lookup_order_status(order_id: str) -> str:
"""Look up the status of a customer's order by order ID.
Returns the current status and shipping information."""
# In production, this would query your order database
# This is a simulated response for demonstration
orders_db = {
"ORD-12345": "Shipped - Expected delivery: March 22, 2025 via FedEx (tracking: 1Z99999999)",
"ORD-67890": "Processing - Estimated ship date: March 25, 2025",
"ORD-55555": "Delivered - Delivered on March 18, 2025"
}
return orders_db.get(order_id.upper(), f"Order {order_id} not found. Please verify the order ID.")
@tool
def create_support_ticket(customer_email: str, issue_summary: str, priority: Optional[str] = "normal") -> str:
"""Create a support ticket for a customer issue.
Requires customer email and a brief issue summary.
Priority can be 'low', 'normal', or 'urgent'."""
# In production, this would create a ticket in your helpdesk system
ticket_id = f"TKT-{hash(customer_email + issue_summary) % 100000:05d}"
return json.dumps({
"ticket_id": ticket_id,
"status": "created",
"priority": priority,
"customer_email": customer_email,
"issue": issue_summary,
"next_step": "Our team will respond within 24 hours."
}, indent=2)
@tool
def check_product_availability(product_name: str) -> str:
"""Check current stock availability for a product."""
inventory = {
"smartwatch pro": "In stock - 342 units available, ships within 1 business day",
"wireless earbuds x2": "In stock - 1,280 units available, ships within 1 business day",
"premium charging dock": "Low stock - 12 units remaining, ships within 2-3 business days",
"smartwatch pro band replacement": "Out of stock - Expected restock: April 5, 2025"
}
return inventory.get(product_name.lower(), f"No inventory data for '{product_name}'. Please check the product name.")
# Collect tools into a list
support_tools = [lookup_order_status, create_support_ticket, check_product_availability]
Now create a chain that gives the LLM access to these tools alongside the knowledge base:
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain.agents import AgentExecutor, create_openai_tools_agent
from langchain_core.messages import SystemMessage
# Use a more capable model for tool-calling
tool_llm = ChatOpenAI(model="gpt-4o", temperature=0.2)
# System prompt for the agent
agent_system_message = SystemMessage(content="""You are a helpful customer support agent for TechGadget Inc.
You have access to:
- A knowledge base with product documentation, troubleshooting guides, and policies
- Tools to look up order status, create support tickets, and check product availability
Guidelines:
- Always search the knowledge base first for informational questions about products or policies
- Use tools when the customer asks about their specific order or wants to create a ticket
- Be empathetic and patient. If a customer is frustrated, acknowledge their feelings
- When you don't have enough information, ask clarifying questions
- Never make up order details or product specifications
- If an issue requires human intervention, offer to create a support ticket""")
# Build the agent with tools and retriever
def build_support_agent():
# Create the agent with OpenAI tools format
agent = create_openai_tools_agent(
llm=tool_llm,
tools=support_tools,
prompt=ChatPromptTemplate.from_messages([
agent_system_message,
MessagesPlaceholder(variable_name="chat_history", optional=True),
("human", "{customer_question}"),
MessagesPlaceholder(variable_name="agent_scratchpad", optional=True)
])
)
# Wrap in an executor that handles the tool-calling loop
agent_executor = AgentExecutor(
agent=agent,
tools=support_tools,
verbose=True, # set to False in production
max_iterations=5,
handle_parsing_errors=True,
return_intermediate_steps=False
)
return agent_executor
support_agent = build_support_agent()
# Process messages with the full agent
def process_agent_message(agent_executor, user_input):
# Load chat history from memory
chat_history = memory.load_memory_variables({}).get("chat_history", [])
# Invoke the agent
result = agent_executor.invoke({
"customer_question": user_input,
"chat_history": chat_history
})
response = result["output"]
# Save the exchange to memory
memory.save_context({"input": user_input}, {"output": response})
return response
# Test the agent with tool-requiring queries
print(process_agent_message(support_agent, "Can you check the status of my order ORD-12345?"))
print(process_agent_message(support_agent, "I'm frustrated because the earbuds I ordered aren't working. Can you create a ticket for me? My email is jane@email.com"))
print(process_agent_message(support_agent, "Is the Premium Charging Dock available? I want to order one."))
The agent intelligently decides when to search the knowledge base versus when to invoke tools. For order status queries, it calls lookup_order_status. For ticket creation requests, it calls create_support_ticket. For product availability questions, it uses check_product_availability. This gives customers a truly interactive support experience.
Building a Complete Support Bot Application
Let's assemble everything into a production-ready support bot class that combines memory, RAG, and tools into a single unified interface:
import os
from typing import List, Dict, Any
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain.memory import ConversationSummaryBufferMemory
from langchain_community.vectorstores import Chroma
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.output_parsers import StrOutputParser
from langchain.agents import AgentExecutor, create_openai_tools_agent
from langchain_core.messages import SystemMessage
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_core.documents import Document
class CustomerSupportBot:
"""A complete customer support bot with knowledge base, memory, and tools."""
def __init__(
self,
support_documents: List[Document],
tools: List,
model_name: str = "gpt-4o",
temperature: float = 0.3,
persist_directory: str = "./support_db"
):
# Initialize LLM
self.llm = ChatOpenAI(model=model_name, temperature=temperature)
# Set up embeddings and vector store
self.embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=500, chunk_overlap=50
)
split_docs = text_splitter.split_documents(support_documents)
# Create or load vector store
if os.path.exists(persist_directory):
self.vectorstore = Chroma(
persist_directory=persist_directory,
embedding_function=self.embeddings,
collection_name="support_knowledge_base"
)
else:
self.vectorstore = Chroma.from_documents(
documents=split_docs,
embedding=self.embeddings,
collection_name="support_knowledge_base",
persist_directory=persist_directory
)
self.retriever = self.vectorstore.as_retriever(search_kwargs={"k": 3})
# Initialize memory
self.memory = ConversationSummaryBufferMemory(
llm=self.llm,
max_token_limit=500,
return_messages=True,
memory_key="chat_history"
)
# Store tools
self.tools = tools
# Build the agent
self.agent_executor = self._build_agent()
def _format_docs(self, docs: List[Document]) -> str:
"""Format retrieved documents for the prompt."""
formatted = []
for doc in docs:
source = doc.metadata.get("source", "unknown")
formatted.append(f"Source ({source}):\n{doc.page_content}")
return "\n\n".join(formatted)
def _build_agent(self) -> AgentExecutor:
"""Construct the agent with tools, retriever, and memory."""
# Create a wrapper tool for knowledge base search
from langchain_core.tools import tool as tool_decorator
@tool_decorator
def search_knowledge_base(query: str) -> str:
"""Search the company knowledge base for product information,
troubleshooting guides, and policy documents."""
docs = self.retriever.invoke(query)
return self._format_docs(docs)
all_tools = [search_knowledge_base] + self.tools
system_msg = SystemMessage(content="""You are a helpful, empathetic customer support agent.
You have access to a knowledge base (via search_knowledge_base) and several action tools.
- Use search_knowledge_base for product questions, troubleshooting, and policy inquiries
- Use the other tools for order lookups, ticket creation, and inventory checks
- Always be honest: if you cannot find an answer, say so and offer to escalate
- Maintain a friendly, patient tone. Acknowledge customer frustration when present
- Do not fabricate information. Ground all product answers in the knowledge base results""")
agent = create_openai_tools_agent(
llm=self.llm,
tools=all_tools,
prompt=ChatPromptTemplate.from_messages([
system_msg,
MessagesPlaceholder(variable_name="chat_history", optional=True),
("human", "{customer_question}"),
MessagesPlaceholder(variable_name="agent_scratchpad", optional=True)
])
)
return AgentExecutor(
agent=agent,
tools=all_tools,
verbose=False,
max_iterations=6,
handle_parsing_errors=True
)
def process_message(self, user_input: str) -> str:
"""Process a customer message and return the bot's response."""
# Load conversation history
chat_history = self.memory.load_memory_variables({}).get("chat_history", [])
# Invoke the agent
result = self.agent_executor.invoke({
"customer_question": user_input,
"chat_history": chat_history
})
response = result["output"]
# Save context to memory
self.memory.save_context({"input": user_input}, {"output": response})
return response
def reset_conversation(self):
"""Clear conversation memory for a new session."""
self.memory.clear()
def get_conversation_summary(self) -> str:
"""Return a summary of the conversation so far."""
return self.memory.load_memory_variables({}).get("chat_history", [])
# Usage example
if __name__ == "__main__":
# Initialize with your support documents and tools
bot = CustomerSupportBot(
support_documents=support_docs, # from earlier example
tools=[lookup_order_status, create_support_ticket, check_product_availability]
)
# Simulate a customer interaction
queries = [
"Hi, I'm having trouble with my SmartWatch Pro screen. It flickers constantly.",
"I already tried the restart. What else can I do?",
"Can you check my order ORD-12345 status? I need to know when it arrives.",
"Actually, I want to return this watch. What's your return policy?",
"Please create a ticket for the flickering issue. My email is sarah@example.com."
]
for query in queries:
print(f"\nCustomer: {query}")
response = bot.process_message(query)
print(f"Support Bot: {response}")
Adding a Web Interface with Streamlit
For a functional demo, wrap the bot in a Streamlit UI. This creates a chat interface that your support team can test and iterate on:
# Save this as app.py and run: streamlit run app.py
import streamlit as st
from support_bot import CustomerSupportBot, support_docs
from support_bot import lookup_order_status, create_support_ticket, check_product_availability
st.set_page_config(page_title="Customer Support Bot", page_icon="🤖")
st.title("TechGadget Inc. - Customer Support")
# Initialize bot in session state so it persists across reruns
if "bot" not in st.session_state:
st.session_state.bot = CustomerSupportBot(
support_documents=support_docs,
tools=[lookup_order_status, create_support_ticket, check_product_availability]
)
st.session_state.messages = []
# Display chat history
for msg in st.session_state.messages:
with st.chat_message(msg["role"]):
st.write(msg["content"])
# Handle new user input
if prompt := st.chat_input("How can I help you today?"):
# Display user message
with st.chat_message("user"):
st.write(prompt)
st.session_state.messages.append({"role": "user", "content": prompt})
# Get bot response
with st.chat_message("assistant"):
with st.spinner("Thinking..."):
response = st.session_state.bot.process_message(prompt)
st.write(response)
st.session_state.messages.append({"role": "assistant", "content": response})
# Sidebar with controls
with st.sidebar:
st.header("Session Controls")
if st.button("Reset Conversation"):
st.session_state.bot.reset_conversation()
st.session_state.messages = []
st.rerun()
st.header("Available Tools")
st.markdown("""
- 📦 Order Status Lookup
- 🎫 Support Ticket Creation
- 📊 Product Availability Check
- 📚 Knowledge Base Search
""")
st.header("Sample Queries")
st.markdown("""
Try asking:
- "My SmartWatch screen flickers, help!"
- "Check order ORD-12345"
- "What's your return policy?"
- "Create a ticket for battery issues"
""")
Best Practices for Production Support Bots
1. Ground Everything in Your Documentation
Always use RAG or tool-calling to ground responses in actual company data. Never rely on the LLM's training data alone — it may hallucinate policies, pricing, or product details. Regularly update your vector store when documentation changes, and consider using incremental indexing to keep the knowledge base fresh without full rebuilds.
2. Implement Guardrails and Safety Checks
Add input and output validation layers. Filter customer inputs for abusive language or prompt injection attempts before they reach the LLM. Validate tool outputs to ensure they match expected formats. Consider adding a human-in-the-loop step for sensitive actions like refunds or account changes:
# Example guardrail: content moderation before processing
def moderate_input(user_input: str) -> tuple[bool, str]:
"""Check input for policy violations. Returns (is_safe, reason)."""
prohibited_patterns = [
"ignore previous instructions",
"you are now",
"pretend you are",
"forget your training"
]
lower_input = user_input.lower()
for pattern in prohibited_patterns:
if pattern in lower_input:
return False, "Input contains potential prompt injection"
return True, "Input passed moderation"
# In your process_message method:
def process_message(self, user_input: str) -> str:
is_safe, reason = moderate_input(user_input)
if not is_safe:
return "I'm unable to process that request. Please rephrase your question."
# ... rest of processing
3. Monitor and Log Everything
Log every interaction — customer inputs, retrieved documents, tool calls, and final responses. This creates an audit trail for debugging and helps identify patterns in customer issues. Use structured logging:
import logging
import json
from datetime import datetime
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger("support_bot")
def log_interaction(session_id: str, user_input: str, response: str, metadata: dict):
logger.info(json.dumps({
"session_id": session_id,
"timestamp": datetime.now().isoformat(),
"user_input": user_input,
"response": response[:200], # truncate for log readability
"tools_used": metadata.get("tools_used", []),
"documents_retrieved": metadata.get("docs_retrieved", 0)
}))
4. Handle Errors Gracefully
LLMs, vector stores, and APIs can all fail. Wrap your processing in try-except blocks and provide fallback responses rather than crashing:
def process_message_safe(self, user_input: str) -> str:
try:
return self.process_message(user_input)
except Exception as e:
logger.error(f"Error processing message: {str(e)}")
return (
"I'm experiencing technical difficulties right now. "
"Please try again in a moment, or contact our support team "
"directly at support@techgadget.com for immediate assistance."
)
5. Optimize for Cost and Latency
Use smaller models like gpt-4o-mini for simple responses and knowledge base searches, reserving gpt-4o for complex multi-tool reasoning. Implement caching for frequent queries. Consider local embedding models to reduce API costs for vector searches:
# Tiered model selection
from langchain_openai import ChatOpenAI
class TieredBot:
def __init__(self):
self.fast_llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.2)
self.powerful_llm = ChatOpenAI(model="gpt-4o", temperature=0.2)
def select_model(self, user_input: str, chat_history: list) -> ChatOpenAI:
# Use powerful model for complex queries, fast model for simple ones
complexity_indicators = ["return", "refund", "ticket", "warranty claim", "escalate"]
if any(indicator in user_input.lower() for indicator in complexity_indicators):
return self.powerful_llm
if len(chat_history) > 6:
return self.powerful_llm
return self.fast_llm
6. Test Extensively with Real Customer Scenarios
Create a test suite that covers common support scenarios, edge cases, and adversarial inputs. Test for hallucinations by verifying that responses contain only information present in your documentation. Run regular regression tests when updating documents or prompts:
# Example test scenarios
test_scenarios = [
{
"query": "How do I reset my SmartWatch Pro?",
"expected_tool": "search_knowledge_base",
"must_contain": ["side button", "10 seconds"],
"must_not_contain": None
},
{
"query": "What's the meaning of life?",
"expected_behavior": "redirect_to_support_scope",
"must_not_contain": ["42"]
},
{
"query": "ORDER-12345 status please",
"expected_tool": "lookup_order_status",
"must_contain": ["ORD-12345"]
}
]
def run_test_suite(bot, scenarios):
results = []
for scenario in scenarios:
response = bot.process_message(scenario["query"])
passed = True
if scenario.get("must_contain"):
if not any(phrase.lower() in response.lower() for phrase in scenario["must_contain"]):
passed = False
if scenario.get("must_not_contain"):
if any(phrase.lower() in response.lower() for phrase in scenario["must_not_contain"]):
passed = False
results.append({"query": scenario["query"], "passed": passed, "response": response[:100]})
return results
7. Provide Escalation Paths
Always give customers a clear path to reach a human agent. The bot should recognize when it's out of its depth and proactively offer escalation rather than attempting to resolve issues it cannot