# Speech-to-Speech Models in Voice AI: Where They Win—and Where They Don’t Speech-to-Speech models deliver natural, low-latency conversations, but enterprise Voice AI needs more than natural speech. Here’s where S2S fits—and where it doesn’t. # Speech-to-Speech Models in Voice AI: Where They Win—and Where They Don’t Speech-to-Speech models are changing how Voice AI feels. Instead of processing a conversation through a traditional pipeline: **Speech → STT → LLM → TTS → Speech** an S2S model can take audio directly and produce audio directly. That has an obvious advantage. The model can preserve things that disappear at the transcript boundary: - hesitation - emotion - rhythm - tone - overlapping speech - mid-sentence code-switching It can also deliver very fast, natural voice-to-voice interaction. But there is an architectural trade-off. > **S2S collapses the three places where a cascaded system can intervene—after transcription, before tool execution, and before rendering—into one probabilistic sampling step.** That difference matters enormously in enterprise Voice AI. --- ## Where the Market Stands As of September 2026, both architectures are valid—but for different use cases. ### Cascaded Voice AI **STT → LLM → TTS** remains the stronger production architecture for phone-based enterprise applications because it offers: - lower cost - swappable components - deterministic tool calling - prompt caching - clearer control points ### Speech-to-Speech S2S is production-viable particularly for **web and in-app experiences** where prosody, emotion and natural reactivity matter more. Current examples referenced in the market include: - OpenAI GPT-Realtime-2 - Gemini 3.1 Flash Live - Amazon Nova 2 Sonic - Kyutai Moshi - Ultravox The key question is therefore not: > **Is S2S better than cascaded Voice AI?** It is: > **Which architecture fits the job the agent actually needs to perform?** --- # The Core Enterprise Problem: No Text Boundary Means No Policy Boundary In a cascaded architecture, the transcript is a first-class object. That means the system can: - inspect the transcript - calculate STT confidence - run rules - apply intent classifiers - validate slots - refuse execution if confidence is too low For example, if the customer says: > “My account number ends in 4827.” a cascaded system can verify whether those digits were captured confidently before proceeding. With pure S2S, the model’s interpretation of the audio does not necessarily surface as an inspectable artifact before action. You may receive: **a tool call** or **an audio response** and the argument binding itself is generated rather than independently extracted. For values such as: - account numbers - Aadhaar last four digits - amounts - dates this becomes a serious limitation unless another ASR layer is running in parallel. At that point, the architecture has effectively introduced a cascade again. --- # Verbatim Output Is Harder Many enterprise conversations contain phrases that must be spoken exactly. Examples include: - regulatory disclosures - consent statements - eligibility statements - mandatory scripts In an S2S model, the output is generated probabilistically. The model may be instructed to follow a script, but each response is still sampled. The problem is not the obvious failure. The problem is the small paraphrase that happens on one call out of thousands. A cascaded architecture can bypass reasoning entirely: Approved Template ↓ TTS ↓ `Exact Spoken Statement` This gives you something fundamentally different from: > “The model usually follows the script.” It gives you an execution path that is structurally unable to rewrite the approved text. --- # Auditability Is Also Different A cascaded Voice AI system can produce a trace such as: Audio ↓ Transcript ↓ Decision ↓ Tool Action ↓ Rendered Text ↓ `Audio` This makes it possible to inspect what happened on each turn. With S2S, the native path is closer to: Audio In ↓ Model ↓ `Audio Out` A transcript produced afterward may not represent exactly what the model acted upon during execution. That makes detailed decision tracing more difficult. For regulated or high-value enterprise workflows, this matters. --- # Tool Calls Create Another Problem Imagine the Voice AI agent needs to update a CRM and the API takes three seconds. What should the S2S model do during those three seconds? It can: - remain silent - generate conversational filler - continue speaking Each option introduces a trade-off. A cascaded runtime controls this explicitly. It can play a deterministic phrase such as: > “Let me check that for you.” while the tool executes. The same issue appears during interruptions. A cascaded runtime can mark a mandatory disclosure as non-interruptible. In many S2S architectures, interruption behaviour is more global. --- # Testing Becomes Harder A cascaded architecture lets engineering teams test individual layers. You can independently test: **STT** **intent extraction** **workflow transitions** **tool calls** **rendered response** **TTS** With S2S, the test often becomes: Audio In ↓ Entire Model ↓ `Audio Out` That means a model-version change can affect: - perception - reasoning - response generation at the same time. For enterprise Voice AI, evaluation also needs to verify more than whether a conversation sounded good. The system may need to prove that the correct **database state** resulted from the call. --- # Telephony Changes the Equation S2S performs best when it receives rich audio. Web and in-app audio can operate at higher quality. PSTN telephony commonly provides **8 kHz audio**, which reduces the quality available to the model. At the same time, S2S pricing is generally based on audio tokens in both directions. For repetitive, high-volume phone workflows, the cascaded model therefore remains attractive from both a control and cost perspective. Language also matters. With a cascade, the platform can choose different STT models for different languages rather than being limited to the language capabilities of a single S2S provider. --- # Where S2S Is Actually Strong This does not mean S2S is the wrong architecture. There are use cases where it is extremely compelling. The common characteristic is that **how something is said contains as much information as what is said**. --- ## Language Learning and Pronunciation Coaching The system needs to hear: - pronunciation - accent - stress - rhythm - hesitation A transcript removes much of the information the product needs. S2S is naturally suited to this. --- ## Companion and Coaching Experiences Here the experience depends on the system noticing: - hesitation - emotional tone - long pauses - changes in voice Responding with the appropriate tone is central to the product. --- ## Live Interpretation S2S can preserve more of the original speaker’s: - urgency - tone - humour - sarcasm when moving between languages. A cascaded transcription-based system may flatten those signals. --- ## In-App and In-Car Assistants For low-stakes requests such as: > “Skip this song.” or > “What is that building?” fast, natural interaction matters. If the agent misunderstands something, the user can simply repeat it. These experiences also use higher-quality audio than typical PSTN calls. --- ## Gaming and Interactive Characters For gaming, NPCs and interactive storytelling: - interruption - laughter - spontaneous reactions - non-deterministic responses can actually improve the experience. Here, variability is not necessarily a failure. It can be part of the product. --- ## Accessibility S2S can also be useful where conventional ASR struggles with: - speech differences - dysarthria - heavy code-switching An audio-native model may degrade differently from an ASR system that produces an incorrect but confident transcript. --- # A Particularly Interesting Enterprise Use Case: Front-of-Call Triage There is also a useful role for S2S inside enterprise Voice AI. Consider the first few seconds of a support call. The objective may simply be to understand: - why the customer is calling - emotional state - urgency - conversational cues S2S can help here. But once the conversation moves into: - identity verification - transactions - CRM writes - regulated disclosures control can move to a deterministic workflow. This means S2S can become a **perception layer**, without becoming the actor responsible for enterprise execution. --- # The Common Pattern The strongest S2S use cases generally share three characteristics: ### Paralinguistic information matters Tone, emotion or speech behaviour carries important information. ### There is no critical transactional write The conversation is not directly committing a high-risk business action. ### A wrong turn is recoverable If the system misunderstands something, the user can repeat or correct it. Invert those conditions and deterministic execution becomes much more important. --- # Where Pure S2S Is Harder to Use The architecture becomes more challenging in environments such as: - banking - telecom - insurance - government services - transactional customer service because these conversations frequently involve: **slot filling** **identity verification** **CRM writes** **amounts** **dates** **mandatory scripts** **auditability** A demo can sound extremely natural. Production requirements are different. --- # The Hybrid Architecture This leads to a more practical architecture: Voice Input ↓ Perception / S2S Emotion · Prosody Turn-taking hints ↓ Deterministic Runtime ↓ Policy + Workflow ↓ LLM when reasoning is required ↓ Validated Execution ↓ ` Voice Output` S2S does not need to replace the entire Voice AI architecture. It can strengthen particular layers of it. --- # How VARTA Thinks About S2S For VARTA, S2S fits naturally as a capability inside a layered architecture rather than as the entire platform. It can contribute to areas such as: **VIRAM-style empathy** **fast perception** **turn-taking** **emotional signals** while the deterministic execution layers continue to own: - policies - business rules - tool execution - workflow state - auditability - rendering guarantees This also keeps the architecture open to future improvements. As S2S models improve, better perception can be introduced without giving up deterministic enterprise controls. That is an important distinction. # S2S can make Voice AI more human. # It does not automatically make Voice AI more reliable. For production systems, both matter. And the strongest architecture may be the one that knows **when to use each.** --- **VARTA Engineering** *The execution layer for production-grade Voice AI.* VARTA’s layered architecture is designed so emerging Speech-to-Speech capabilities can be introduced where natural perception, prosody and turn-taking add value, while deterministic layers continue to control business execution.