Voice AI Latency Is a Systems Problem, Not a Model Problem

A Voice AI agent can use an extremely fast language model and still feel slow.

That is because the customer does not experience LLM latency.

They experience the time between:

finishing a sentence and hearing the agent respond.

That delay is produced by an entire runtime pipeline.

A simplified Voice AI path looks like this:

Audio → Endpoint Detection → STT → Intent / Routing → LLM or Workflow → APIs / Tools → TTS → Telephony

If you want to reduce Voice AI latency, optimizing one component in isolation is rarely enough.

You need to optimize the critical path.

Start With a Latency Budget

Instead of asking:

“How fast is our Voice AI?”

break the response into measurable components.

StageWhat to Measure

Endpointing

Time between user finishing and end-of-turn detection

STT

Time until a stable transcript is available

Routing

Time to identify intent, workflow and next action

LLM

Time to first token when reasoning is required

Tool Call

CRM, ERP or third-party API response time

TTS

Time to first playable audio

Telephony

Audio delivery and buffering overhead

The key user-facing metric is:

End of User Speech → First Agent Audio

A model may return its first token in 500 milliseconds while the caller still waits several seconds before hearing anything.

That is the difference between model latency and conversation latency.

1. Latency Starts Before the LLM

Before the system can process what the caller said, it first has to determine whether the caller has finished speaking.

Consider:

“Book the appointment for Friday…”

A short pause.

“…after 3 PM.”

If the runtime waits too long, the agent feels slow.

If it commits too quickly, it interrupts the customer.

This is the endpointing problem.

A production Voice AI system should not depend only on a fixed silence timer.

It can also consider:

• voice activity
• transcript stability
• sentence completion
• current workflow step
• expected response type

For example, if the agent asks:

“Would you prefer Monday or Tuesday?”

and receives a stable:

“Tuesday.”

the runtime may already have enough information to continue.

The correct endpointing behaviour can therefore depend on the conversation state.

2. Not Every Turn Needs an LLM

Consider a simple interaction:

Agent: Which day works better, Monday or Tuesday?

Customer: Tuesday.

The workflow already knows:

• what question was asked
• what inputs are valid
• what the customer selected
• what step comes next

Sending the entire conversation to an LLM to understand the word “Tuesday” introduces unnecessary inference and network latency.

A better runtime decision looks like this:

Can this turn be resolved deterministically?

YES NO
| |
Workflow / Rules LLM Reasoning
| |
+----------+------------+
|
Next Action

LLMs are extremely useful when a caller moves outside the expected path.

For example:

“Before booking, can you tell me whether my product is still under warranty?”

That requires interpretation and possibly knowledge retrieval.

But many transactional turns are predictable:

• yes / no
• date selection
• time slot selection
• OTP collection
• ratings
• consent
• callback confirmation
• repeat requests

Using deterministic execution for these turns can improve both speed and reliability.

3. Conversation State Should Not Live Only Inside the Prompt

Long calls create another performance problem: context growth.

A model request may eventually include:

• system instructions
• business rules
• conversation history
• customer details
• retrieved knowledge
• previous API responses
• workflow instructions

Yet the current turn may simply be:

“Yes.”

If the system is currently confirming an appointment, most of that historical context may be unnecessary.

Production systems should maintain explicit conversation state separately.

For example:

intent: service_booking
step: confirm_date
customer_id: C10291
proposed_date: 2026-09-11
warranty_status: active

This means the runtime does not need to reconstruct application state from natural language on every turn.

It also provides:

• lower token processing
• more predictable behaviour
• better debugging
• lower latency

Conversation history is useful for understanding language.

It should not be the application's only state store.

4. APIs Can Be Slower Than AI

Suppose a customer asks:

“What slots are available tomorrow?”

The Voice AI system may understand the request immediately.

But now it needs information from:

• CRM
• scheduling engine
• field service system
• ERP
• technician availability
• another enterprise API

The slowest component in a production call may not be AI at all.

It may be an existing business system.

That means tool execution needs to be included in the latency budget.

Where possible, the system can acknowledge immediately:

“Let me check the available slots.”

while the API request executes.

This allows speech playback and backend processing to overlap.

A useful engineering principle is:

Don't make the caller wait for operations that can run in parallel.

5. TTS Should Not Always Be Generated From Scratch

Enterprise calls contain a large amount of repeated speech.

Examples include:

“Thank you.”

“One moment while I check.”

“Could you confirm your registered mobile number?”

“Your appointment has been confirmed.”

If a sentence has already been approved and spoken thousands of times, generating it from scratch on every call adds unnecessary work.

A production Voice AI runtime can separate responses into:

Static responses
Pre-generated and stored.

Cacheable responses
Generated once and reused when appropriate.

Dynamic responses
Generated in real time.

The runtime can then follow a simple path:

Response Required
|
v
Cached audio available?
/ \
YES NO
| |
Play Generate TTS

Caching helps with cost.

More importantly for latency, it can reduce time to first audio dramatically on common conversation turns.

6. Optimize for First Audio

Another common mistake is waiting for the entire response to be generated before speech begins.

Imagine the agent needs to say:

“Your warranty is active until December, and I can schedule a technician for Friday afternoon.”

A sequential pipeline might:

1. generate the complete text
2. generate the complete audio
3. start playback

A streaming pipeline can instead operate like this:

Model Output
↓
Text Chunks
↓
Streaming TTS
↓
Audio Frames
↓
Caller

The caller starts hearing the beginning of the response while later parts are still being generated.

This is why time to first audio is often a more meaningful metric than total response-generation time.

Architecture Matters More Than a Single Benchmark

Voice AI vendors often describe latency using a number such as:

“Our model responds in X milliseconds.”

That number may be useful.

But it does not tell you how long the caller waits.

A better engineering question is:

What is on the critical path between end-of-user-speech and first-agent-audio?

Then optimize that path.

That may involve:

• better endpointing
• streaming STT
• deterministic routing
• fewer LLM calls
• smaller context windows
• parallel API execution
• cached audio
• streaming TTS

Only one of those optimizations is:

Use a faster model.

How VARTA Thinks About Latency

At VARTA, the core design principle is:

Don't route every turn through the most expensive reasoning path.

A Voice AI runtime should distinguish between:

known workflow transitions

and

genuinely ambiguous conversations.

Known transitions can follow deterministic execution paths.

Reusable speech can come from intelligent caches.

Business information can come directly from enterprise systems.

LLMs can be invoked when deeper interpretation or reasoning is actually required.

And each layer should be observable independently so engineering teams can answer:

Where exactly did this turn spend its time?

Because the goal is not to build the fastest LLM integration.

The goal is to build a faster conversation.

The Metric That Matters

For every production call, measure:

User End-of-Speech → Agent First Audio

Then break that number down by layer.

Once Voice AI teams start thinking this way, latency stops being a vague AI problem.

It becomes what it really is:

A systems engineering problem.

VARTA Engineering

Building the execution layer for production-grade Voice AI.