Highlights

Ultra-low latency: VoxServe is built for real-time speech, delivering extremely low inference latency. In the demo below, a TTS request achieves a Time-To-First-Audio (TTFA) as low as 40 ms on an NVIDIA H100 GPU.

Real-time LLM chat integration: Qwen3-TTS supports incremental text input, and VoxServe supports that capability, making it easy to build end-to-end voice chatbots. The video below shows VoxServe connected to a local LLM, achieving low end-to-end response latency.

High throughput: VoxServe is optimized not just for low latency, but also for high-throughput serving under load. The figure below compares streaming performance against vLLM-Omni (v0.14.0) across increasing request rates. The y-axis reports TTFA (time to first audio chunk). The annotation boxes report streaming viability, i.e., the fraction of audio chunks delivered in time to avoid playback gaps on the client. All experiments follow the setup in our paper. The benchmark script is available here.

Serving performance.

While vLLM-Omni supports online serving for Qwen3-TTS, it does not currently support streaming audio generation, which keeps TTFA high even at low request rates. VoxServe treats streaming generation as a first-class objective, enabling effective batching while keeping TTFA low even under heavy concurrency.

Usage

Install VoxServe and serve a Qwen3-TTS checkpoint:

pip install vox-serve
vox-serve --model Qwen/Qwen3-TTS-12Hz-1.7B-Base --port 8000

Generate speech with a simple curl request:

curl -X POST "http://localhost:8000/generate" \
  -F "text=Hello, this is a demonstration of Qwen3-TTS served by VoxServe." \
  -F "streaming=true" \
  -o output.wav

For detailed examples, see the Qwen3-TTS usage page:

• https://vox-serve.github.io/vox-serve/usage/qwen3-tts.html

We also provide an interactive playground for quick experimentation.

Technical Details

Qwen3-TTS is a state-of-the-art text-to-speech model from Alibaba’s Qwen team. It delivers strong audio quality, but serving it well is non-trivial: the architecture is multi-stage, supports multiple modes (Base, CustomVoice, VoiceDesign), and requires careful input/output streaming for low-latency inference.

VoxServe is a high-efficiency serving system built specifically for speech models. It provides a stable execution abstraction that accommodates a wide range of modern speech architectures, while enabling system-level optimizations like continuous batching, cache management, and CUDA Graph execution. As with many other models in our ecosystem, VoxServe supports the full Qwen3-TTS feature set with low latency in streaming scenarios.

Below, we outline how VoxServe maps cleanly onto Qwen3-TTS.

Model Architecture

Model architecture of Qwen3-TTS.

Qwen3-TTS is composed of four major components:

1. Speech Encoder: optionally encodes reference speech for voice cloning (Base variant)
2. Qwen3 LM (Talker): generates speech tokens for codebook 0
3. MTP Module (Codec Predictor): generates speech tokens for codebooks 1–15
4. Streaming Codec Decoder: converts 16 codebooks into waveform audio

Three components (talker, codec predictor, and codec decoder) operate autoregressively. That creates engineering challenges around request scheduling, cache management, and GPU utilization. The codec decoder also includes audio-specific operations (e.g., convolutions) that introduce additional state to manage for streaming. Finally, the three Qwen3-TTS variants require different input configurations, adding more surface area to the serving stack.

Despite this complexity, the model fits naturally into VoxServe’s execution interface.

System design of VoxServe.

VoxServe implements a shared execution pipeline for all the models:

Preprocess → LM Forward → Sampling (→ Depth Forward → Depth Sampling) → Postprocess

• Preprocess: Formats inputs and runs the speech encoder when needed. Qwen3-TTS inputs vary by variant: speaker IDs for CustomVoice, reference audio/text for Base voice cloning, and instruction-style prompts for VoiceDesign.
• LM Forward & Sampling: Runs the talker (Qwen3 LM). Each step consumes a single text token plus 16 audio tokens, the content of which varies depending on whether input streaming is enabled, and voice cloning can additionally inject audio feature vectors. VoxServe’s interface supports this cleanly via three buffers: input_ids, input_masks, and input_features. We did not need to change this interface to support the full functionality of the Qwen3-TTS model.
• Depth Forward & Sampling: Runs the codec predictor (MTP). VoxServe already supports this class of “depth” modules (e.g., in CSM-1B), so Qwen3-TTS plugs into an existing interface.
• Postprocess: Runs the codec decoder and emits waveform audio. For streaming, this stage requires cache management for both attention and convolutional layers. VoxServe already handled detokenizer caching for other models (e.g., CosyVoice 2); enabling it for Qwen3-TTS required just defining a new cache class and wiring the decoder to read/write cache state.

The key point is that we support Qwen3-TTS’s model-specific details without materially changing the layers above, such as the scheduler and worker in the diagram. That design choice matters because it lets existing system optimizations – continuous batching, KV/detokenizer cache management, CUDA Graph execution, and scheduling policies – apply to Qwen3-TTS with minimal friction. This is especially important for speech models, where architectures vary significantly across families. For deeper detail on the interface and optimizations, see our paper.

Input Streaming Implementation

VoxServe supports Qwen3-TTS’s incremental text input feature through a custom scheduler implementation.

VoxServe’s scheduler is explicitly designed to expose performance optimization hooks. In our paper, we describe streaming-oriented scheduling policies that prioritize low TTFA while preserving output streamability. This is implemented via request selection logic applied at each scheduler iteration.

Input streaming can be seamlessly supported in a similar way by swapping in a different scheduler implementation. The main challenge arises when audio generation outpaces incoming text; VoxServe addresses this by employing request selection rules that manage partially available input, ensuring the output stream remains smooth and uninterrupted.

Try VoxServe

VoxServe is fully open source on GitHub: https://github.com/vox-serve/vox-serve

Give it a try and let us know what you think!

Code is open-sourced here: https://github.com/vox-serve/vox-serve

In recent years, Speech Language Models (SpeechLMs), such as Text-to-Speech (TTS) and Speech-to-Speech (STS) models built on Language Model (LM) backbones, have gained significant traction. The release of powerful open-source models is opening up exciting opportunities for speech AI applications.

However, deploying these models in practice remains challenging:

1. Lack of standardized abstractions. SpeechLMs vary widely in architecture, and there is no common framework to unify inference across them. This makes it difficult to switch between models.
2. Limited focus on efficiency. To our knowledge, no inference system exists that is designed specifically for SpeechLMs with an emphasis on low-latency, high-throughput deployment. As a result, serving these models can be slow and costly.

In practice, each new speech model often comes with its own custom inference stack, which may not necessarily prioritize efficiency, making it cumbersome to switch between models. Repurposing existing LLM serving systems also demands significant effort due to fundamental differences in architecture and inference algorithms.

VoxServe addresses these challenges by providing a unified interface that supports diverse SpeechLMs, with high performance as the core design goal.

SpeechLM Background

Modern SpeechLMs typically consist of an LM backbone and an audio detokenizer model: the LM autoregressively generates discrete audio tokens, which the detokenizer then converts into continuous audio data.

Overview of typical SpeechLMs.

Serving these models efficiently poses unique challenges. At every inference step, two different models must run in tandem, while the resulting binary audio data needs to be streamed to the client. To enable stream generation, both models must be carefully scheduled at the right intervals.

Additionally, there are numerous model-specific complexities that complicate implementation, including multi-codebook modeling, depth transformers, audio input encoders, repetition penalties, and watermarking requirements. Audio detokenizers themselves vary widely in architecture, size, and latency characteristics, further increasing the difficulty.

For optimal serving performance, request scheduling must be carefully designed to account for both the LM backbone and the audio detokenizer.

VoxServe

We solve these challenges by designing VoxServe, a new serving system for SpeechLMs from the ground up. VoxServe currently supports the following four models, with more on the way:

• CSM
• Orpheus
• Zonos
• GLM-Voice
• Step-Audio-2-Mini

The VoxServe model class is designed to natively support multi-stream inference, a common requirement for speech workloads. Its abstraction is carefully engineered to strike a balance between flexibility and efficiency, making it compatible with batch inference, CUDA graphs, and streaming.

A typical inference pipeline in VoxServe includes:

• Preprocessing: Preparing inputs for the LM backbone, such as prompt formatting, encoder inference, and metadata or masking setup for sampling.
• LM forward: Running the LM backbone to generate logits for the next tokens.
• Sampling: Selecting the next tokens from logits, which may involve algorithms like repetition penalty, classifier-free guidance, or filtering based on token type (e.g., audio vs. text tokens).
• (Optional) Depth forward: Executing the depth transformer for models that autoregressively generate tokens across multiple codebooks.
• Postprocessing: Converting tokens into audio data using the detokenizer, with a unified interface across diverse architectures.

By standardizing this workflow while remaining adaptable to model-specific variations, VoxServe simplifies deployment and ensures performance across a wide range of SpeechLMs.

Performance Optimizations

VoxServe goes beyond basic model inference, introducing optimizations specifically tailored to SpeechLMs in order to maximize performance.

Scheduling Algorithm

Since SpeechLMs comprise multiple components (the LM backbone and the audio detokenizer), the scheduling of requests between them has a direct impact on performance. Importantly, we note that speech applications differ in their performance requirements, so VoxServe implements specialized scheduling strategies for two distinct scenarios: online serving and offline serving.

Online serving scenarios: For interactive applications like voice chatbots, where many requests arrive in random intervals, we define the following two metrics:

• Time-To-First-Audio (TTFA): The latency from user input to the first audio chunk. Unlike Time-To-First-Token (TTFT) in LLMs, this requires generating multiple tokens and running the detokenizer (and sometimes the encoder) before producing the first chunk.
• Streaming Viability: Once the first audio chunk is ready, subsequent audio must be generated faster than the playback speed to prevent audio disruption that the client experiences.

A notable difference from text generation is that speed improvements beyond playback rate have diminishing returns (except for the first chunk), i.e., as long as the generation satisfies the real-time requirements, there is no benefit in generating faster than that. This opens the door to a scheduling strategy that prioritizes requests only when they are critical (either because the first audio chunk has not yet been produced or because streaming viability is at risk).

VoxServe classifies requests as critical or non-critical based on their current progress. Critical requests are prioritized, while non-critical ones can be delayed slightly to improve overall hardware utilization without hurting latency or streaming quality.

Intuitively, you can delay the inference of some part of the model for better hardware utilization, as long as it affects neither TTFA nor streaming viability.

Examples of online scheduling optimizations.

Offline serving scenarios: On the other hand, for workloads such as audiobook or podcast generation, the priority shifts from latency to throughput.

For offline serving, the performance metric is end-to-end throughput, which we measure by the Real-Time Factor (RTF), i.e., the total length of generated audio divided by the time it takes to generate it.

The scheduling strategy is simpler: maximize throughput by keeping hardware fully utilized, typically by running large batches at each stage (LM backbone and detokenizer).

Asynchronous Execution

To minimize overhead from complex scheduling and metadata processing, VoxServe adopts an asynchronous execution pipeline. Both the LM backbone and the audio detokenizer run asynchronously with respect to their schedulers, leveraging a delayed stop-decision mechanism (as proposed in NanoFlow).

Pipeline of asynchronous execution.

Evaluation

We evaluate the performance of VoxServe on Orpheus-3B model on a single H100 GPU. As a baseline, we compare against the official implementation provided by the model developers, which uses vLLM for LM backbone inference. All evaluations use greedy sampling with no repetition penalty.

For online serving, we measure TTFA and streaming viability rate (the fraction of audio chunks meeting real-time playback requirements) under varying request arrival rates (modeled as a Poisson distribution, with each request generating 1024 tokens).

The baseline system shows long TTFA and poor streaming viability, even under light loads. In contrast, VoxServe maintains low TTFA and meets real-time playback requirements thanks to its optimized detokenizer implementation, batching strategies, and use of CUDA graphs. By adopting the optimized scheduling algorithm, TTFA is kept low with an even higher request rate.

Performance for online serving scenario.

VoxServe achieves better throughput for the offline serving scenario as well. In an experiment processing 100 requests of equal length (1024 tokens), VoxServe significantly outperforms the baseline, owing to its coordinated scheduling of the LM backbone and audio detokenizer. With optimized scheduling enabled, throughput improves by an additional ~15%.

Performance for offline serving scenario.

What’s Next?

We are actively working on supporting more models and further performance improvements.

If you’d like to try it out, you can install VoxServe with:

pip install vox-serve

Also, please check out the code and feel free to post any requests or bug reports at our GitHub Issues.