Knowledge assistant over 12k docs

Challenge

Knowledge locked in thousands of documents

The team's best answers lived in PDFs, wikis and shared drives. Finding the right clause or precedent meant asking a colleague or searching for an hour — and new hires were slowest of all.

Objectives

What we set out to achieve

Give everyone instant, trustworthy answers grounded in the firm's own material.

• 90%+Answer accuracy on a test set
• <5sAverage response time
• 0Data leaving approved systems

Strategy

Start with the questions people actually ask

We collected 300 real questions, mapped them to source documents and built an evaluation set before writing a line of code — so quality could be measured, not guessed.

Solution

Grounded answers with citations

A RAG pipeline with hybrid search, permission-aware retrieval and an assistant that cites the exact passage it used — and says "I don't know" when the sources are silent.

Technology

The stack behind it

Chosen for reliability, speed and easy ownership by the client team.

• ClClaude
• OpOpenAI
• LaLangChain
• pgpgvector
• PyPython
• FaFastAPI
• AzAzure
• LaLangSmith

Implementation

10 weeks, four phases

Weekly demos kept the client team involved at every step.

1. Weeks 1–2Discovery & evaluation setQuestion mining, source audit, access rules.
2. Weeks 3–6Ingestion & retrievalChunking, embeddings, hybrid search, permissions.
3. Weeks 7–8Assistant & guardrailsCitations, refusals, tone and formatting.
4. Weeks 9–10Pilot & rolloutPilot group, feedback loop, firm-wide launch.

Results

Answers in seconds, not hours

94% Accuracy within the first quarter, and accuracy keeps improving as the evaluation set grows.