Speech-to-Speech Models in Voice AI: Where They Win—and Where They Don’t

Speech-to-Speech models are changing how Voice AI feels.

Instead of processing a conversation through a traditional pipeline:

Speech → STT → LLM → TTS → Speech

an S2S model can take audio directly and produce audio directly.

That has an obvious advantage.

The model can preserve things that disappear at the transcript boundary:

• hesitation
• emotion
• rhythm
• tone
• overlapping speech
• mid-sentence code-switching

It can also deliver very fast, natural voice-to-voice interaction.

But there is an architectural trade-off.

S2S collapses the three places where a cascaded system can intervene—after transcription, before tool execution, and before rendering—into one probabilistic sampling step.

That difference matters enormously in enterprise Voice AI.

Where the Market Stands

As of September 2026, both architectures are valid—but for different use cases.

Cascaded Voice AI

STT → LLM → TTS

remains the stronger production architecture for phone-based enterprise applications because it offers:

• lower cost
• swappable components
• deterministic tool calling
• prompt caching
• clearer control points

Speech-to-Speech

S2S is production-viable particularly for web and in-app experiences where prosody, emotion and natural reactivity matter more.

Current examples referenced in the market include:

• OpenAI GPT-Realtime-2
• Gemini 3.1 Flash Live
• Amazon Nova 2 Sonic
• Kyutai Moshi
• Ultravox

The key question is therefore not:

Is S2S better than cascaded Voice AI?

It is:

Which architecture fits the job the agent actually needs to perform?

The Core Enterprise Problem: No Text Boundary Means No Policy Boundary

In a cascaded architecture, the transcript is a first-class object.

That means the system can:

• inspect the transcript
• calculate STT confidence
• run rules
• apply intent classifiers
• validate slots
• refuse execution if confidence is too low

For example, if the customer says:

“My account number ends in 4827.”

a cascaded system can verify whether those digits were captured confidently before proceeding.

With pure S2S, the model’s interpretation of the audio does not necessarily surface as an inspectable artifact before action.

You may receive:

a tool call

or

an audio response

and the argument binding itself is generated rather than independently extracted.

For values such as:

• account numbers
• Aadhaar last four digits
• amounts
• dates

this becomes a serious limitation unless another ASR layer is running in parallel.

At that point, the architecture has effectively introduced a cascade again.

Verbatim Output Is Harder

Many enterprise conversations contain phrases that must be spoken exactly.

Examples include:

• regulatory disclosures
• consent statements
• eligibility statements
• mandatory scripts

In an S2S model, the output is generated probabilistically.

The model may be instructed to follow a script, but each response is still sampled.

The problem is not the obvious failure.

The problem is the small paraphrase that happens on one call out of thousands.

A cascaded architecture can bypass reasoning entirely:

Approved Template
↓
TTS
↓
Exact Spoken Statement

This gives you something fundamentally different from:

“The model usually follows the script.”

It gives you an execution path that is structurally unable to rewrite the approved text.

Auditability Is Also Different

A cascaded Voice AI system can produce a trace such as:

Audio
↓
Transcript
↓
Decision
↓
Tool Action
↓
Rendered Text
↓
Audio

This makes it possible to inspect what happened on each turn.

With S2S, the native path is closer to:

Audio In
↓
Model
↓
Audio Out

A transcript produced afterward may not represent exactly what the model acted upon during execution.

That makes detailed decision tracing more difficult.

For regulated or high-value enterprise workflows, this matters.

Tool Calls Create Another Problem

Imagine the Voice AI agent needs to update a CRM and the API takes three seconds.

What should the S2S model do during those three seconds?

It can:

• remain silent
• generate conversational filler
• continue speaking

Each option introduces a trade-off.

A cascaded runtime controls this explicitly.

It can play a deterministic phrase such as:

“Let me check that for you.”

while the tool executes.

The same issue appears during interruptions.

A cascaded runtime can mark a mandatory disclosure as non-interruptible.

In many S2S architectures, interruption behaviour is more global.

Testing Becomes Harder

A cascaded architecture lets engineering teams test individual layers.

You can independently test:

STT

intent extraction

workflow transitions

tool calls

rendered response

TTS

With S2S, the test often becomes:

Audio In
↓
Entire Model
↓
Audio Out

That means a model-version change can affect:

• perception
• reasoning
• response generation

at the same time.

For enterprise Voice AI, evaluation also needs to verify more than whether a conversation sounded good.

The system may need to prove that the correct database state resulted from the call.

Telephony Changes the Equation

S2S performs best when it receives rich audio.

Web and in-app audio can operate at higher quality.

PSTN telephony commonly provides 8 kHz audio, which reduces the quality available to the model.

At the same time, S2S pricing is generally based on audio tokens in both directions.

For repetitive, high-volume phone workflows, the cascaded model therefore remains attractive from both a control and cost perspective.

Language also matters.

With a cascade, the platform can choose different STT models for different languages rather than being limited to the language capabilities of a single S2S provider.

Where S2S Is Actually Strong

This does not mean S2S is the wrong architecture.

There are use cases where it is extremely compelling.

The common characteristic is that how something is said contains as much information as what is said.

Language Learning and Pronunciation Coaching

The system needs to hear:

• pronunciation
• accent
• stress
• rhythm
• hesitation

A transcript removes much of the information the product needs.

S2S is naturally suited to this.

Companion and Coaching Experiences

Here the experience depends on the system noticing:

• hesitation
• emotional tone
• long pauses
• changes in voice

Responding with the appropriate tone is central to the product.

Live Interpretation

S2S can preserve more of the original speaker’s:

• urgency
• tone
• humour
• sarcasm

when moving between languages.

A cascaded transcription-based system may flatten those signals.

In-App and In-Car Assistants

For low-stakes requests such as:

“Skip this song.”

or

“What is that building?”

fast, natural interaction matters.

If the agent misunderstands something, the user can simply repeat it.

These experiences also use higher-quality audio than typical PSTN calls.

Gaming and Interactive Characters

For gaming, NPCs and interactive storytelling:

• interruption
• laughter
• spontaneous reactions
• non-deterministic responses

can actually improve the experience.

Here, variability is not necessarily a failure.

It can be part of the product.

Accessibility

S2S can also be useful where conventional ASR struggles with:

• speech differences
• dysarthria
• heavy code-switching

An audio-native model may degrade differently from an ASR system that produces an incorrect but confident transcript.

A Particularly Interesting Enterprise Use Case: Front-of-Call Triage

There is also a useful role for S2S inside enterprise Voice AI.

Consider the first few seconds of a support call.

The objective may simply be to understand:

• why the customer is calling
• emotional state
• urgency
• conversational cues

S2S can help here.

But once the conversation moves into:

• identity verification
• transactions
• CRM writes
• regulated disclosures

control can move to a deterministic workflow.

This means S2S can become a perception layer, without becoming the actor responsible for enterprise execution.

The Common Pattern

The strongest S2S use cases generally share three characteristics:

Paralinguistic information matters

Tone, emotion or speech behaviour carries important information.

There is no critical transactional write

The conversation is not directly committing a high-risk business action.

A wrong turn is recoverable

If the system misunderstands something, the user can repeat or correct it.

Invert those conditions and deterministic execution becomes much more important.

Where Pure S2S Is Harder to Use

The architecture becomes more challenging in environments such as:

• banking
• telecom
• insurance
• government services
• transactional customer service

because these conversations frequently involve:

slot filling

identity verification

CRM writes

amounts

dates

mandatory scripts

auditability

A demo can sound extremely natural.

Production requirements are different.

The Hybrid Architecture

This leads to a more practical architecture:

Voice Input
↓
Perception / S2S
Emotion · Prosody
Turn-taking hints
↓
Deterministic Runtime
↓
Policy + Workflow
↓
LLM when reasoning
is required
↓
Validated Execution
↓
Voice Output

S2S does not need to replace the entire Voice AI architecture.

It can strengthen particular layers of it.

How VARTA Thinks About S2S

For VARTA, S2S fits naturally as a capability inside a layered architecture rather than as the entire platform.

It can contribute to areas such as:

VIRAM-style empathy

fast perception

turn-taking

emotional signals

while the deterministic execution layers continue to own:

• policies
• business rules
• tool execution
• workflow state
• auditability
• rendering guarantees

This also keeps the architecture open to future improvements.

As S2S models improve, better perception can be introduced without giving up deterministic enterprise controls.

That is an important distinction.

S2S can make Voice AI more human.

It does not automatically make Voice AI more reliable.

For production systems, both matter.

And the strongest architecture may be the one that knows when to use each.

VARTA Engineering

The execution layer for production-grade Voice AI.

VARTA’s layered architecture is designed so emerging Speech-to-Speech capabilities can be introduced where natural perception, prosody and turn-taking add value, while deterministic layers continue to control business execution.