Why Doesn't Your Company Chatbot Know Its Own Leave Policy?

You ask your company's Slack bot: "Can I carry forward my unused sick leaves to next year?"

It replies, instantly, confidently: "Yes, up to 5 days."

Sounds right. Except your company's actual policy, updated last quarter, caps carry-forward at 3 days. The bot didn't lie on purpose. It just doesn't know what it doesn't know.

---

The problem with a plain LLM

A base LLM, GPT, Claude, whatever, is trained on a massive slice of the internet. Books, articles, code, forums. What it isn't trained on: your company's internal HR wiki, updated last Tuesday by someone in People Ops.

So when you ask it something specific to your organization, it has two options. Say "I don't know." Or generate the most statistically plausible answer based on patterns from similar policies it's seen elsewhere. Most models, tuned to be helpful, lean toward option two. That's how you get a confident, well-written, completely made-up number.

This isn't a knowledge problem you can fix by making the model bigger. It's a data access problem.

Enter RAG

Retrieval-Augmented Generation solves this by changing what happens right before the model answers.

Instead of relying purely on what it memorized during training, the system first searches your actual knowledge base (your leave policy doc, your product manual, your support tickets) for the chunks most relevant to the question. Those chunks get inserted into the prompt as context. Only then does the model generate a response, grounded in real, current text instead of a statistical guess.

Your query. A search over real documents. The relevant pieces get handed to the model. The model answers using what it was just shown.

Same question, run through this pipeline: the system pulls the actual leave policy paragraph, the model reads "carry-forward capped at 3 days, revised FY24," and answers correctly. No guessing required.

Where RAG genuinely works well

When the source document is clear, current, and directly answers the question, RAG is excellent. Internal knowledge bases, product documentation bots, legal and compliance Q&A, customer support over a help center. Anywhere the ground truth lives in text the model can retrieve, RAG turns a generic chatbot into one that actually knows your stuff.

But "grounded in retrieved text" isn't the same as "guaranteed correct." RAG reduces one category of error. It introduces several new ones.

---

When retrieval itself goes wrong

The whole pipeline depends on step one: finding the right chunk. If retrieval pulls the wrong one, everything downstream inherits that mistake.

Say your knowledge base also has a maternity leave policy and a work-from-home policy. If the retrieval system matches on surface-level word overlap instead of actual intent, it might hand the model a chunk about maternity leave carry-forward instead of sick leave. The model, working with what it was given, answers confidently anyway. Wrong chunk in, wrong answer out. The model isn't hallucinating here, it's doing exactly what it was told to do with bad material.

Poor chunking makes it worse

Even when retrieval grabs the right document, how that document got split into chunks matters enormously.

Good chunking respects natural boundaries: a full policy point stays together, self-contained. Poor chunking often just slices text every N characters, no regard for sentence or paragraph structure. If "carry-forward is capped at 3 days" gets sliced in half across two separate chunks during indexing, and only one half gets retrieved, the model receives an incomplete sentence and fills the gap itself. Usually wrong.

Context window limits

Even with several good chunks retrieved, there's a ceiling on how much text fits into the model's context window at once. Stuff in too many chunks, and the most relevant one might get buried or trimmed to make room. Models also tend to pay less attention to information sitting in the middle of a long context, a well-documented effect. More retrieved context isn't automatically better. Precision beats volume.

Hallucination doesn't disappear

Here's the part people underestimate: even with the exact right chunk retrieved and fully in context, the model can still get it wrong. It might misread a number, blend the retrieved fact with something it remembers from training, or paraphrase in a way that quietly changes the meaning. RAG reduces hallucination. It does not eliminate it. The model is still a text generator, not a database lookup, even when you hand it the right page.

Stale knowledge bases

RAG is only as current as its index. If the HR team updates the policy PDF but nobody re-runs the indexing pipeline, the retrieval system keeps serving the old version, confidently, with a straight face. This is an easy failure to miss because nothing looks broken. The bot answers fast, cites something real, and it's simply out of date.

When RAG isn't the right tool

Some questions don't belong to RAG at all. "How many sick leaves do I personally have left?" needs a live lookup against an HR system's database, not a static policy document. That's a job for a tool call or API integration, not retrieval. Highly structured data (tables, exact figures, anything better suited to a SQL query) also tends to perform poorly through text retrieval. And for anything legally or financially binding, a grounded answer still deserves a human check before it goes out the door.

---

Back to that Slack message

Done right, the fix for that wrong carry-forward answer isn't a smarter model. It's a properly indexed, well-chunked, regularly updated knowledge base feeding the right paragraph to the model at the right time. Get that pipeline right, and "up to 5 days" becomes "up to 3 days, per the policy revised this quarter," with a source to back it up.

RAG doesn't make a model omniscient. It makes it accountable to something real, as long as you keep that "something real" accurate.

What's the strangest wrong answer a RAG-powered bot has ever given you? Drop it below.