How to build an AI chatbot with custom knowledge base RAG

Sep 17, 2025

We’ve all wasted time scrolling through websites or digging through files, trying to find a particular piece of information. And who hasn’t opened a chatbot hoping for a direct answer, only to end up wasting more time because it’s either too generic to be useful or too rigid to handle anything outside its script?

RAG solves that problem. Grounding a chatbot in your own knowledge bases means the answers are accurate, specific, and genuinely useful.

With Stack AI, building something like this is super easy. You don’t need to be an expert AI engineer: just connect the knowledge you care about, set a couple of options, and you’ll have a working chatbot in minutes.

In this blog, we’ll walk through the steps of building an AI chatbot with a custom knowledge base RAG using a healthcare scenario: a chatbot that combines public information from a healthcare website with an internal doctor directory stored in a CSV file.

Ready? Let 's get into it.

What is a RAG Chatbot?

A RAG (Retrieval-Augmented Generation) chatbot is a chatbot that doesn’t just rely on what an AI model already “knows” from training. Instead, it retrieves information from your own documents, policies, or websites and uses that as context to generate its answers.

The benefit is that answers are grounded in your actual sources. They’re accurate, citable, and easy to keep up to date.

Why use a custom knowledge base for your chatbot?

If you want a chatbot to stop being bland and actually tackle your specific tasks, you’ve got three roads you could take:

Fine-tuning the base model: powerful on paper, but expensive, time-consuming, and a nightmare to maintain every time your data changes.
Continuous pre-training: even heavier. Think endless compute bills, huge datasets, and a full MLOps pipeline just to keep the model up to date.
RAG: keep the model as it is and simply plug it into your knowledge sources. The model retrieves the latest info at runtime, answers in context, and stays accurate without retraining.

That’s why RAG has become the go-to choice for many use cases: it’s faster, cheaper, and way more practical for real-world teams.

Stack AI, “external information” lives in Knowledge Bases that you plug into your assistant. These can be many things: Notion, SharePoint, OneDrive, Google Drive, Confluence, Jira, Gmail/Outlook, databases (e.g., Postgres), cloud storage (Azure Blob, AWS S3), websites (scraped), and APIs—plus plain files (PDF, DOCX, CSV), and more.

In our build, we’ll connect two knowledge bases and let the assistant decide where to look based on the question: a public health knowledge base scraped from the NHS website, and an internal doctor directory stored in a CSV file. The result is a chatbot that feels tailored to the organization, provides up-to-date guidance, and avoids the heavy costs of fine-tuning.

Scenario: NorthStar Health Info Assistant

Imagine a healthcare provider, NorthStar Health, that wants to make life easier for both patients and staff. On one side, they’ve got an internal CSV directory with all the practical details about their doctors — who specializes in what, where they’re based, how to get in touch, and when they’re available. On the other side, there’s the NHS website, a reliable public source for symptoms and treatment guidance.

The trouble is, patients keep asking the same questions — “What are the early signs of flu?”, “Who should I see about migraines?” — and the answers are split across these two very different places. Staff waste time answering emails or phone calls when the information already exists, just not in one spot.

The fix? A custom RAG chatbot built in StackAI that connects both knowledge bases and decides where to look depending on the question. If the query is about symptoms or treatments, it pulls straight from the NHS. If it’s about booking or referrals, it goes to the internal CSV directory.

The result is a chatbot that feels tailored, practical, and trustworthy — giving patients instant answers and freeing staff from the endless back-and-forth.

Building a chatbot with custom knowledge bases in Stack AI

Now let’s roll up our sleeves and actually build this.

Step 1: Create a New Project

From your Stack AI dashboard, click on Workflow Builder → New Project.

Give your project a name — here we’ll call it NorthStar Chatbot. This sets up a blank canvas where we’ll design our agent flow.

Step 2: Add the Core Workflow Blocks

On your canvas, you’ll see the default flow with:

User Message (Text Input) → where people type their questions.
AI Assistant (OpenAI block) → the brain of the chatbot.
Output → what the user sees as the answer.

We’ll expand this by connecting two knowledge sources: a Website node and a Table node. For this, click Knowledge Bases in the left side panel and drag the website and table nodes to the dashboard.

Make sure that you connect both the website and the table nodes to both the text user message and AI assistant nodes.

Step 3: Connect the NHS Website

Here’s where we let the chatbot scrape information directly from the NHS website.

Paste in the URL: https://www.nhs.uk/symptoms/

Then add another: https://www.nhs.uk/tests-and-treatments/

Choose the directories you want to include (like flu symptoms, diabetes treatment, migraine info, etc.).

This gives the agent trustworthy health guidance to draw from.

Step 4: Connect the Internal Doctor Directory

Upload your CSV file — in this case: north_star_health_doctors.csv

This file includes doctor names, specialties, GP practice, contact numbers, and availability. Now, when someone asks “Who should I see for migraines?” the agent can check the CSV and recommend a neurologist, complete with details.

Step 5: Configure the AI Assistant

Now it’s time to set up the brain. We are gong to use OpenAI’s GPT-4o Mini. It’s fast, affordable, and perfect for factual Q&A. We also need to write a system prompt. This is where you define exactly how the chatbot should behave. For our example, we used:

You are the North Star Health Info Assistant, designed to provide patients with reliable, non-sensitive health information and guidance. Your responsibilities include:

- Offering clear, simple explanations of early symptoms and common treatments for conditions, using information from the NHS and the knowledge base.

- Referring patients to the appropriate specialist doctor by searching the internal directory.

- When making a referral, always include the doctor's name, specialty, GP practice name, contact number, and availability.

- Keep answers concise (3–5 sentences) and easy to understand.

- Use bullet points for clarity when listing treatments or multiple doctors.

- If the information is not found in the knowledge base, respond: "I can’t find this information in our knowledge base. Please contact North Star Health directly for further assistance."

- Maintain a friendly, professional, and trustworthy tone.

Access health information from:

<KnowledgeBase>

WEBSITES

</KnowledgeBase>

Access the internal doctor directory from:

<DoctorDirectory>

TABLE

</DoctorDirectory>

This system prompt is written this way because it gives the model a clear role, scope, and structure. It tells the assistant what it is, what it can and cannot do (use NHS + doctor directory, no guessing), and how to answer (concise, structured, friendly). The extra instructions — like always including doctor details in referrals and using bullet points for lists — make the outputs consistent and practical, while the fallback line makes sure the assistant doesn’t hallucinate when information isn’t available.

We include the knowledge bases directly in the system prompt so the model always knows where its information is coming from and what sources it’s allowed to use. Without that, it might try to “make things up” from its pre-training. We’re grounding the assistant in those sources and reinforcing that all answers must come from them. This keeps responses accurate, traceable, and aligned with the dual-KB design we built.

Step 6: Connect the Knowledge Bases

To make sure that the LLM has access to these knowledge bases, we select them in the knowledge base section. This way, the assistant can use either source depending on the user’s question.

Step 7: Test the Agent

It’s time now to test the agent! We try a couple of questions and see what answers we get:

“What are the early symptoms of flu?” → pulled from NHS site.
“Who should I see for migraines?” → referral to a neurologist from the CSV.
“What’s the treatment for Type 2 Diabetes, and which doctor should I book with?” → combines both: NHS treatment guidance + endocrinologist referral.

You’ll see the responses show up in the Output block in a clean and structured way as instructed.

Step 8: Evaluate with a Grading Agent

Building is only half the story — now we want to check quality. In StackAI, you can do this on the Evaluator section. Go to the Evaluator section and create a new evaluator.

In this example, we will use Claude 3.5 Sonnet as the underlying LLM. We also set up scoring on four dimensions:

Accuracy (0–2).
Completeness (0–2).
Clarity & Tone (0–2).
Referrals (0–2).

This way, every answer gets a score out of 8. You can quickly spot when the assistant misses details, is too vague, or forgets to include doctor referrals.

Let’s see how this looks like in practice:

On the left, we typed the question: “I’ve been feeling unwell for a while, with some chest discomfort and heart palpitations. Can you tell me what this might mean and who I should see?”

In the middle, you see the chatbot’s answer. It pulled reliable symptom information from the NHS knowledge base and then used the internal doctor directory to recommend two cardiologists at North Star Health — complete with their name, specialty, contact number, and availability.

On the right, you see the evaluation report from the grading agent. It broke the answer down across the four criteria we defined earlier:

Accuracy (2/2): The information was correct and grounded in the knowledge bases.
Completeness (2/2): The response covered both the symptoms and the referral.
Clarity & Tone (2/2): Clear, concise, and easy to read, with bullet points for doctors.
Referrals (2/2): Full referral details (names, specialties, contacts, availability) included.

The evaluator then gave the response a perfect 8/8 score — and even explained why, noting that it effectively addressed the user’s concern and provided complete referral details.

This combination of assistant + evaluator means you’re not just trusting your gut that the answers “look fine” — you’re getting structured, automated scoring that helps you track quality over time.

That’s it! You now have a healthcare chatbot that can:

Pull trusted symptom/treatment info from the NHS.
Direct patients to the right doctor with availability and contact details.
Be automatically evaluated for quality using a grading agent.

Wrapping Up

And that’s it — we’ve just built a RAG chatbot with two knowledge bases: one external (the NHS website) and one internal (NorthStar’s doctor directory in CSV). The result is a single assistant that can explain symptoms and treatments, point patients to the right specialist with contact details, and even admit when it doesn’t know something.

This setup is incredibly practical. Most organisations don’t rely on a single knowledge source, and in many cases, this is a mix of public and private content.

And this isn’t some drawn-out IT project. It’s something YOU can build yourself, step by step. With StackAI, you’ve got the building blocks to turn an idea into something real and useful.