Engineering

LLMs, LangChain, Guardrails: The Trifecta of Advanced Chatbots

Naresh Reddy, Praveen Kumar, Praveen JU, Naresh Reddy N, Ajit Narayanan7 min read5 March 2025

Authors: Praveen Kumar, Praveen JU, Naresh Reddy N, Ajit Narayanan

Animated illustration of a user and a chatbot exchanging messages.

In today’s highly competitive business environment, swiftly addressing customer concerns has become crucial for businesses aiming to maintain customer loyalty and satisfaction. Customer support agents play a pivotal role in this process, as they are often the first point of contact and directly influence the customer’s perception of the brand. Effective customer support not only resolves immediate issues but also strengthens the long-term relationship between customers and businesses.

Chatbots — advanced computer programs specifically designed to mimic human-like interactions — have dramatically transformed the landscape of customer service. By providing round-the-clock assistance without any downtime, chatbots ensure that customers receive immediate responses regardless of the time or location. They are capable of simultaneously managing multiple customer interactions, a feat that is challenging for human agents, thus significantly enhancing operational efficiency.

Furthermore, the integration of chatbots reduces a business’s dependency on extensive human support teams, allowing human agents to focus on complex tasks that require empathy, judgment, and nuanced understanding. This approach not only leads to considerable cost savings but also ensures consistency in responses, minimizes human error, and ultimately boosts overall customer satisfaction.

Chatbots are broadly classified into two categories:

Guided Flow based Chatbots:

The customer queries are restricted to a predefined set of questions along with the conversation flow. The restriction results in delayed response and feels unnatural to use. The guided flow based chatbot at Licious is shown in figure below

Licious Help menu offering support for delayed orders, missing or incorrect items, product quality and refunds.

2. Artificial Intelligence based Chatbot:

The customers can enter their query as a free text in the provided text box and the trained chatbot provides a response. The interactions between AI chatbot and user feel more natural and customers get quick resolution to their concerns. The AI based chatbots are leveraged from the advancements in the field of Natural Language Processing (NLP) starting from N-Gram models, RNNs, LSTMs to Transformer models. The AI based chatbots can be of two types:

a.) Machine Learning (ML) powered chatbot:

The customer query (input text) is comprehended to understand the intent, sentiment, and then generate text. A small dataset is collected to train a separated model for each of the above mentioned tasks. For instance, BERT is used for NER, classifiers such as SVM are used for Sentiment Analysis and sentence-to-sentence models such as LSTMs are used for text generation.

Natural language processing combines language understanding, machine learning and language generation to answer a customer’s return-policy question.

b.) Large Language Model (LLMs) powered chatbot: LLMs are a significant leap forward in generative AI technology and are notable for their ability to achieve general purpose natural language processing and generation. Leveraging onto the strong ability of LLMs natural language understanding, chatbots can be designed to engage in conversations that are complex and require deeper understanding of context.

Context Windows in LLM-powered Chatbots:

Context windows define how much prior conversation history an LLM can process in a single interaction. The effectiveness of an LLM-powered chatbot significantly depends on how well it manages its context window.

Short Context Windows: Some LLMs process a limited number of tokens at a time, which can lead to loss of previous conversation history. This makes it challenging for chatbots to maintain long-running conversations and provide coherent responses.
Long Context Windows: Advanced LLMs, like GPT-4 and Claude-2, support longer context windows, allowing chatbots to remember and respond based on earlier parts of the conversation. This enhances user experience by maintaining conversation continuity and coherence.
Sliding Context Windows: To optimize performance, chatbots use techniques such as summarization and relevance-based retrieval to keep only the most important details within the context window.

The performance of ML powered chatbots depends on the performance of multiple NLP models that are tied together. Further, the ML powered chatbot can easily break down while interacting with customers for reasons that might sound silly to humans. For example, use of abbreviations, misspelled words, slangs, accents/dialects etc. On the other hand, LLM powered chatbots leverage the vast knowledge base, logical reasoning and text generation capabilities to make the interaction with customers feel more meaningful, context-aware, natural and intuitive. The requirement of minimal instructions (i.e., prompts) to tune the behavior of LLMs is icing on the cake.

The performance of the AI chatbots is largely dependent on the performance for the generative language model used. Large Language Models (LLMs) are the most powerful generative language models till date and with their advent, the landscape of AI based chatbot is set to change significantly.

LLM Powered Chatbot Architecture:

Leveraging Large Language Models (LLMs) for chatbot development is an exciting opportunity, but it also comes with significant responsibility — after all, “With great power comes great responsibility.” This article explores the ethical and responsible use of LLMs in building chatbots, enabling organizations to enhance customer experiences and unlock new business opportunities.

Some of the most advanced LLMs available today include OpenAI’s GPT-3.5 and GPT-4, Google’s PaLM and Gemini, Meta’s LLaMA series, Anthropic’s Claude series, xAI’s Grok, and Mistral AI’s models. In this article, we utilized OpenAI’s GPT-3.5-turbo to develop a customer support chatbot. The block diagram of the proposed method is shown in the figure below.

Enterprise chatbot architecture with NeMo Guardrails, optional retrieval and memory, and a language model generating an answer.

Guardrails for AI Safety:

LLMs can sometimes generate inaccurate or irrelevant responses. While prompt engineering improves chatbot safety, it is crucial to implement additional safeguards. Guardrails act as protective layers, preventing chatbots from engaging in harmful topics or responding inaccurately.

In this work, we integrated Nvidia’s NeMo Guardrails as a filtering layer. This ensures that:

Harmful or off-topic queries are either filtered or responded to with predefined safe replies.
Company-specific guidelines are adhered to, restricting discussions to relevant business topics.
Conversational flows are controlled using canonical representations of user queries.

LangChain for Retrieval-Augmented Generation (RAG):

LLMs rely on static pre-trained knowledge and may lack real-time information. For customer support chatbots, integrating a dynamic knowledge retrieval system is crucial. Retrieval-Augmented Generation (RAG) bridges this gap by allowing chatbots to fetch relevant information from external knowledge bases. Steps included are shown in the below diagram:

Retrieval-augmented generation pipeline: load documents, split and store them, retrieve relevant passages, and pass the context and prompt to a language model.

LangChain provides a framework to implement RAG by:

Loading external documents such as FAQs, product catalogs, and support guides.
Splitting documents into smaller chunks and embedding them in a vector database.
Using a retriever module to fetch the most relevant information based on similarity scores.
Augmenting chatbot responses with real-time retrieved information.

Example Query Processing with RAG

User Query: What hygiene steps do you follow?

LLM Prompt:

You are a Customer Support Agent for a trusted consumer brand. Use the following pieces of context to answer customer queries. If you don’t know the answer, ask follow-up questions — don’t try to make up answers. Keep the response as concise as possible.
{context}
Question: {Query}
Here, {context} is dynamically populated with relevant information retrieved from the knowledge base.

Optimizing Context Window Utilization in Chatbots

Managing context windows efficiently improves chatbot performance and response accuracy. Some optimization techniques include:

Context Compression: Summarizing previous conversations to retain key details while minimizing token usage.
Relevance-based Retention: Storing only the most relevant historical messages to maximize useful context within the token limit.
Hierarchical Memory Management: Implementing short-term memory for recent exchanges and long-term memory for persistent user preferences.
Token-efficient Formatting: Structuring responses concisely to prevent unnecessary token consumption.

LLM-Powered Chatbot in Action

The chatbot designed for Licious integrates these principles to deliver precise, context-aware customer support. It:

Accurately responds to order-related queries.
Provides up-to-date information on hygiene and food safety.
Rejects off-topic queries while maintaining conversational engagement.

For example, if a user inquires about stock market trends, the chatbot gracefully redirects them to Licious-related discussions instead of generating an irrelevant answer.

Conclusion

The design of LLM-powered customer support chatbots hinges on three core pillars: context window optimization, guardrails for AI safety, and RAG for dynamic knowledge retrieval. By leveraging advanced LLMs in combination with LangChain’s retrieval capabilities and NeMo Guardrails, organizations can create highly effective, context-aware chatbots that streamline customer interactions and improve service quality.