Designing RAG Assistants: What Knowledge They May (and May Not) Use

A Retrieval‑Augmented Generation (RAG) assistant can feel like magic: “Ask me anything about our policies, products, or processes.” Under the hood, it’s just a model plus a retrieval layer over your documents. The risk is simple: if you index the wrong content, the assistant can surface it in convincing ways.

Design is where you prevent that.

What makes RAG different

Unlike pure GenAI, a RAG assistant:

• Relies on a base model for language fluency.
• Pulls facts and details from a separate knowledge base at query time.
• Can be updated by changing the underlying content, not the model.

That means governance is less about model training and more about knowledge scoping and lifecycle.

Define the assistant’s mission and audience

Start your Design with two questions:

• What questions and tasks should this assistant help with?
• Who is it for?

Examples:

• An internal policy assistant for staff.
• A developer documentation assistant for engineers.
• A customer‑facing support assistant for common product questions.

Write this down in one or two sentences. This mission statement will guide every decision that follows.

Decide which knowledge domains are in scope

Based on the mission, select knowledge domains:

• In scope E.g., published product docs, internal FAQs, policy manuals, help centre content, process guides.
• Out of scope E.g., raw legal case files, individual employee records, confidential deal documents, sensitive HR or health records.

Be as concrete as possible:

• “Includes: internal wiki spaces A and B, public docs site, product knowledge base.”
• “Excludes: HR wiki, legal archive, incident reports, personal records.”

Design is the right time to be conservative. You can always expand later.

Classify and tag content for retrieval

DASUD’s Design feeds into how you structure Acquire and Store.

For RAG, ensure that documents are:

• Classified By sensitivity (public, internal, confidential, highly sensitive), domain (HR, product, support), and audience.
• Tagged With metadata like department, product line, region, effective dates, and validity (current vs obsolete).

These tags allow you to:

• Filter out sensitive or obsolete content from retrieval.
• Restrict certain domains to specific user roles.
• Prepare for regional or legal differences in responses.

A RAG system is only as safe as its underlying content classification.

Design behaviour for uncertainty and conflicts

RAG assistants will encounter gaps and conflicts:

• No relevant documents.
• Multiple conflicting documents.
• Outdated or superseded content.

In Design, decide:

• What happens when confidence is low Does the assistant say “I don’t know,” suggest contacting a human, or provide a best‑effort answer with caveats?
• How to surface sources Will you show document titles and links so users can verify? In many governance‑sensitive domains, you should.
• How to handle conflicts Do you prioritise certain sources, favour the latest document by date, or escalate to humans when conflicts are detected?

These choices matter as much as model quality.

Build a RAG design sheet

To make all this practical, create a simple “RAG design sheet” for each assistant:

• Mission and audience.
• In‑scope and out‑of‑scope knowledge domains.
• Approved repositories and their owners.
• Content classification and tagging plan.
• Behaviour for low confidence, missing content, and conflicts.

Share this design sheet with content owners, risk/compliance, and technical teams. It becomes your shared reference point.

Make it concrete

Pick one RAG assistant you have or plan to build. For that assistant:

• Write its mission in one sentence.
• List the repositories you intend to index.
• Mark each as in‑scope or out‑of‑scope.
• Note any content that needs reclassification or tagging before indexing.
• Decide how it should behave when it’s not sure or when content conflicts.

By designing RAG assistants this way, you make knowledge exposure a conscious decision—not an accident of whatever happened to be in your wiki that day.

If you’d like assistance or advice with your Data Governance implementation, or any other topic (Privacy, Cybersecurity, Ethics, AI and Product Management) please feel free to drop me an email here and I will endeavour to get back to you as soon as possible. Alternatively, you can reach out to me on LinkedIn and I will get back to you within the same day!