Case study · Cross-vertical · Knowledge management · 60 days
Same five questions, answered with the source on the page.
A national membership body was repeating the same five answers on every internal call. A retrieval-augmented assistant now grounds every answer in the source policy — with the citation right beside it.
Chapter 01 · The problem
Five questions a day, asked by everyone, answered from memory.
A national membership body had a problem its team would not have called a problem. Member-facing staff were answering roughly five questions a day each — the same five — about eligibility rules, fee waivers, and the documentation required for renewals. The answers were correct, mostly. They came from the heads of three long-tenured staff who had read the policy enough times to recite it. The Knowledge Lead had been quietly mapping which questions came up, who answered them, and how often the answer drifted between people. The drift was the part that worried her.
~250 queries a week
Chapter 02 · The approach
We did not write a chatbot. We put the policy on the page.
The fix was not a separate help system. It was an internal assistant the staff could ask in plain language, which retrieved the relevant policy passage from the SharePoint library and showed the citation beside the answer. The model summarised; it did not interpret. Every answer carried the source paragraph as a click-through. If retrieval found nothing close enough, the assistant said so — and routed the question to the Knowledge Lead’s queue rather than guessing.
What we built
A retrieval pipeline indexing the SharePoint policy library into pgvector, with Anthropic Claude generating grounded summaries against the retrieved passages. n8n handles the SharePoint sync on a nightly cadence. The Hermes orchestrator coordinates the index refresh, the query handler, and the unanswered-question queue.
Build time · 19 days · pilot phase · followed by 6 weeks of iteration with the Knowledge Lead and three member-facing staff.
Chapter 03 · The outcome
Three numbers the Knowledge Lead trusted before she shared them.
- -0%
time-to-first-answer for member queries — measured against the prior quarter average call-handling time.
- 0%
of answers carrying a clickable citation back to the source policy paragraph.
- -0%
drift between staff on the same five questions — measured by the Knowledge Lead’s answer-consistency audit.
Numbers verified by the Knowledge Lead against the call-handling export and her quarterly consistency audit. Anonymised under our standard case-study disclosure: vertical and outcome are real; organisation name, location, and policy-library tenant are not.
Chapter 04 · What we learned
The retrieval was easy. The "I don’t know" was the hard part.
The retrieval pipeline itself took 19 days. The six weeks that followed were spent teaching the assistant to admit when it did not know. The early version would summarise from a tangentially related passage and read fluent — and wrong. We tightened the similarity threshold, added an explicit hand-off prompt for low-confidence retrievals, and changed the answer template so the source paragraph showed first, the summary second.
The lesson was about the staff, not the model. Once the assistant routed unanswered questions back to the Knowledge Lead, she had a clean signal of what the policy library was missing. Twelve passages got rewritten in the first month — not because the model failed, but because the policy itself had been ambiguous and the staff had been quietly papering over it.
Settled handoff rate · 12 policy rewrites
Chapter 05 · In the client’s words
The assistant is helpful, but the real win was finding out which of our policies were quietly broken. We could not see the gaps until we had to answer them in writing.
Knowledge Lead · National membership body · AU (anonymised at the client’s request)
Curious whether your team has the same five questions?
A free 45-minute audit. We look at the questions your team answers most, where the source policy lives, and where the drift is creeping in. You leave with a one-page memo, whether we’d be a fit or not.