Building an Enterprise Knowledge Base with RAG
What we've learned about choosing documents, tuning retrieval, testing answers, and enforcing access controls.

Take inventory before building the index
Loading every document into a vector database at once is a common way to derail a project. Start with one department and a few dozen key documents. Identify who will ask questions and what they need to know before deciding what to index.
Cleaning up documents can take half the work. Remove duplicates, reconcile versions, and extract structured content from tables and images. Those decisions directly affect what the system can retrieve.
Combine keyword and vector search
Vector search alone can miss specialized terms and identifiers. We combine it with keyword search and use a reranker to select the passages most likely to answer the question.
How you split documents matters just as much. Keep paragraphs or sections that cover one topic together, and retain the heading hierarchy so retrieved passages carry their context.
Build an evaluation set before launch
Ask people who know the business to annotate a set of questions and answers that covers common requests and edge cases. Run regression tests against that set every time you change chunking, retrieval, or prompts.
Once the system is live, track retrieval hit rates, clicks on citations, and handoffs to human support. These metrics help you spot gaps in the knowledge base.
Carry over the original permissions
Preserve the access rules from the source documents. Filter results by role and department so sensitive material can only be retrieved by people authorized to see it.
