By Online With AI · Published · 3 min read
A knowledge assistant should help your team find a dependable answer, not turn a messy shared drive into a more confident source of confusion. The difficult work often starts before the model is connected.
What retrieval adds
Instead of relying only on a model’s general training, an application can retrieve relevant company material and provide it as context. Microsoft’s retrieval-augmented generation overview describes this pattern. Retrieval helps bring your information into the answer, but it does not prove that every generated statement is correct.
A practical system should link statements to sources, identify the relevant document version and recognise when the retrieved evidence does not answer the question. The user needs a quick route back to the source, especially when a decision is consequential.
Choose a bounded information set
Start with one collection that has an owner: approved service documentation, operational procedures or current project information. Remove superseded copies or label them clearly. Decide whether informal notes belong in the answer set or should only be used as a lead for further checking.
For a development company, a first assistant might support internal project queries using approved status reports. It should not silently combine a draft assumption with a signed document and call the result a fact. For a recruitment company, separate client information from candidate records and apply the permissions each source requires.
Permissions must survive the search
A person should not be able to retrieve a document through the assistant that they could not otherwise access. Apply permission checks to retrieval and to any stored index. Do not assume that hiding a link in the interface prevents an answer from revealing its contents.
The ICO’s guidance on AI security and data minimisation is a useful starting point for reviewing the information involved. Determine what data the task needs, where it moves and who is responsible for it. Review current requirements for your particular use rather than treating a product feature as a compliance guarantee.
Test answerable and unanswerable questions
Create an evaluation set with straightforward questions, ambiguous terms, conflicting documents and questions outside the approved material. An assistant that always answers can look impressive in a demonstration and be a poor fit for real operations.
Measure whether sources actually support the answer. A citation attached to a paragraph is not sufficient if the document says something different. Also measure how long it takes a user to verify the answer. A system that requires opening ten irrelevant files has not necessarily saved time.
Plan the update process
Agree how new documents enter the system, how withdrawn material disappears and how quickly permissions change. Give users a way to flag an incorrect or stale answer. The owner needs enough context to reproduce the problem without collecting unnecessary personal information.
Start with read-only assistance. Add actions only when there is a clear business case and tested control. The next useful step is an inventory of your approved sources and ten representative questions—not connecting the whole company drive. Explore custom knowledge systems for a scoped implementation.