Building Enterprise Realtime Voice Agents from Scratch: A Technical Tutorial
Jielin Qiu Zixiang Chen Liangwei Yang Ming Zhu Zhiwei Liu Juntao Tan Wenting Zhao Rithesh Murthy Roshan Ram Huan Wang Akshara Prabhakar Shelby Heinecke Caiming Xiong Silvio Savarese
作者信息
Abstract
We present a technical tutorial for building enterprise-grade realtime voice agents from first principles. While end-to-end speech-to-speech models may ultimately provide the best latency for voice agents, fully self-hosted end-to-end solutions are not yet available. We evaluate the closest candidate, Qwen3-Omni, across three configurations: its cloud-only DashScope Realtime API achieves $\sim$702ms audio-to-audio latency with streaming, but is not self-hostable; its local vLLM deployment supports only the Thinker (text generation from audio, 516ms), not the Talker (audio synthesis); and its local Transformers deployment runs the full pipeline but at $\sim$146s -- far too slow for realtime. The cascaded streaming pipeline (STT $\rightarrow$ LLM $\rightarrow$ TTS) therefore remains the practical architecture for self-hosted realtime voice agents, and the focus of this tutorial. We build a complete voice agent using Deepgram (streaming STT), vLLM-served LLMs with function calling (streaming text generation), and ElevenLabs (streaming TTS), achieving a measured time-to-first-audio of 755ms (best case 729ms) with full function calling support. We release the full codebase as a 9-chapter progressive tutorial with working, tested code for every component.引用本文复制引用
Jielin Qiu,Zixiang Chen,Liangwei Yang,Ming Zhu,Zhiwei Liu,Juntao Tan,Wenting Zhao,Rithesh Murthy,Roshan Ram,Huan Wang,Akshara Prabhakar,Shelby Heinecke,Caiming Xiong,Silvio Savarese.Building Enterprise Realtime Voice Agents from Scratch: A Technical Tutorial[EB/OL].(2026-03-17)[2026-10-22].https://arxiv.org/abs/2603.05413.学科分类
计算技术、计算机技术
