A knowledge assistant that answers from your own documents

The challenge

The answers people need already exist. They are in policy documents, manuals, contracts and spreadsheets — and finding them takes long enough that people stop looking and ask a colleague instead.

Keyword search makes this worse rather than better. It returns hundreds of matches ranked by term frequency, which is a useful way to find a document and a useless way to answer a question.

The context

I built this as a product rather than a one-off, which changes the problem. It is not enough for retrieval to work on one carefully prepared corpus. It has to work on whatever a new account uploads on their first afternoon — mixed PDFs, Word files and spreadsheets, inconsistent structure, no clean-up — and it has to do that for many accounts at once without their content ever touching each other.

It also has to be affordable to run continuously, which rules out a lot of architectures that demo well.

The approach

The first decision was what not to build. Fine-tuning a model on the corpus would have been expensive, would have gone stale the moment a document changed, and would have made it impossible to show where an answer came from — which is the property that decides whether anyone trusts the thing.

Retrieval-augmented generation was the right shape: keep the knowledge in an index that updates cheaply, and use the model only for the part it is genuinely good at, which is reading relevant passages and writing a clear answer from them.

The second decision was to treat retrieval quality as the product. Wiring up a vector database is an afternoon. Making it return the right passage on a real corpus is the actual work, and that is where the time went.

The solution

An upload is parsed in a sandbox, then chunked along the document's own heading structure rather than at a fixed character count — roughly 800 tokens with 100 of overlap, carrying the heading path with each chunk so a clause that says "this does not apply in the cases above" is not stranded without its context.

Chunks are embedded by a small multilingual model that runs on CPU, and stored in Postgres with both a vector index and a full-text index. The chunk table is partitioned per account and protected by row-level security, so one account's content is not merely filtered out of another's results — it is not reachable.

A question runs two searches at once: vector similarity and keyword search, forty candidates each. The two ranked lists are fused, and a reranking model scores what comes out and keeps the best six. Only those six reach the language model, which streams back an answer carrying numbered citations that resolve to the exact document and page.

When the retrieved passages do not support an answer, the system says so instead of producing one.

AI & automation

Three models, each doing one job. A small embedding model turns text into vectors. A reranker decides which candidates actually bear on the question. A language model reads the six survivors and writes the answer. None of them is the memory of the system, and none is asked to recall anything.

The interesting engineering sits around them. Hybrid search exists because pure vector search quietly fails on exact terms — a policy number, a product code, a proper noun — returning ten passages about the general topic and missing the one containing the literal string. The reranker exists because semantic similarity and actual relevance are related but not the same thing.

Retrieved passages go into the prompt wrapped in delimiters and labelled as untrusted content, because a document someone uploads is an input, not an instruction. Refusal is specified, prompted for and tested with questions the corpus cannot answer — one confidently invented policy does more damage to adoption than fifty clumsy correct answers.

Engineering

Multi-tenancy was the constraint that shaped the schema. Partitioning chunks by account and enforcing row-level security means isolation is a property of the database rather than a filter somebody has to remember to apply in every query.

Cost drove the model choices. A small embedding model that runs on CPU rather than a GPU-bound one is a deliberate trade: slightly weaker embeddings, recovered by hybrid search and reranking, in exchange for a service that is affordable to leave running.

Parsing files that strangers upload is a security problem, so it happens in a sandbox. Ingestion runs as background jobs through a queue, is idempotent and resumable, so re-processing a changed document does not duplicate it in the index or require a full rebuild — at this volume, a job that cannot restart from where it failed is a job that never finishes.

Answers stream token by token, because a correct answer that appears after eight silent seconds feels broken.

Outcome

People ask a question in plain language and get an answer in seconds with its sources attached, which means they can check it — and being able to check it is what makes them willing to rely on it.

Adding new material is an ingestion run rather than a project.