A voice agent that takes the call, looks it up, and answers

The challenge

A large share of inbound calls are routine, but they do not arrive as tidy form fields. They arrive as somebody talking.

Phone menus handle only the narrowest version of this, and callers dislike them enough to press zero at the first opportunity — which puts the call straight back into the queue it was meant to avoid. The gap was between a menu that cannot understand a sentence and a person who has to be paid to answer the same question all day.

The context

There was already a phone system and there were already backend APIs holding the customer data. Nothing needed inventing at either end; the missing piece was everything in between.

The binding constraint was latency. This is a conversation, not a web form: a pause of a few seconds before a reply reads as a broken line, and the caller starts saying "hello? hello?" over the top of the response. Every architectural decision downstream was shaped by that.

The approach

I built it as a pipeline of single-purpose stages joined by queues rather than as one monolithic bot. Each stage does one job — carry the audio, turn it into text, work out the intent, fetch the data, compose the reply — which means any of them can be scaled, retried or swapped without disturbing the others.

The second principle was to keep the deterministic parts deterministic. Identity and data retrieval are exact operations against systems of record. A model is used where language genuinely needs understanding, and nowhere else. The model never holds customer data; it decides what is being asked, and later phrases what the systems returned.

The solution

The call is bridged out of the phone system and into the application. The audio stream is transcribed as the caller speaks rather than after they finish, so the pipeline is already working while the sentence is still being said.

A language model reads that transcript and works out what is actually being asked. That intent drives a query against the business systems that already hold the answer. A second model turns the result into a natural spoken reply, which goes back down the line.

Queues sit between the stages, so a slow model call or a slow lookup delays a reply rather than dropping a call.

AI & automation

Two distinct model jobs, deliberately kept apart.

The first reads the transcript and decides what the caller wants. It is a classification problem wearing conversational clothes, and treating it as its own stage means it can be evaluated on its own terms — did it identify the right intent — separately from whether the eventual wording was good.

The second takes structured data that the systems returned and turns it into something a person would say out loud. It is given the data and asked to phrase it; it is never asked to know it. That separation is what keeps a voice agent from confidently inventing an account balance.

Engineering

Latency is the whole game, and almost every decision serves it. Transcribing the stream as the caller speaks rather than waiting for a complete utterance removes a delay the caller would otherwise hear. Queues between stages absorb variance instead of propagating it.

Speech is a messier input than text. Transcripts arrive partial, mid-word and occasionally wrong, so the stage boundaries have to tolerate input that no one would have written down.

And because the pipeline spans a phone system, a transcription service, two model calls and a set of internal APIs, each with its own failure mode, the interesting work is in what happens when one of them is slow or unavailable while a human being is holding a live telephone.

Outcome

Callers can ask for what they want in their own words, at any hour, and get an answer drawn from live data rather than a menu tree.

The routine calls stop needing a person, and the ones that genuinely need judgement reach one sooner.