XTTS-v2

XTTS-v2#

XTTS-v2 is Coqui’s multilingual text-to-speech model: a GPT-2 (30 layers) that generates speech codes conditioned on a voice, and a HiFi-GAN vocoder that turns the GPT’s hidden states into 24 kHz audio. It speaks 17 languages and clones a voice from a few seconds of reference audio; 58 voices come with the checkpoint.

The model weights are licensed under the Coqui Public Model License (non-commercial use only).

Quickstart#

vox-serve --model coqui/XTTS-v2 --port 8000   # or --model xtts

curl -X POST localhost:8000/generate -F "text=Hello, this is XTTS speaking." -F "streaming=true" -o output.wav

Request Options#

Field

Default

Description

text

required

Text to synthesize (at most 401 tokens, roughly 250 characters of English; split longer texts)

language

en

en, es, fr, de, it, pt, pl, tr, ru, nl, cs, ar, zh-cn, hu, ko, ja, hi

voice (or speaker)

Claribel Dervla

A built-in voice (the names in the checkpoint’s speakers_xtts.pth, listed in the server’s start-up log)

audio

none

A reference clip to clone (the first 30 s are used); overrides voice

streaming

true

Stream the audio as it is generated

 
import requests

# A built-in voice, in Spanish.
r = requests.post("http://localhost:8000/generate",
                  data={"text": "Hola, ¿cómo estás?", "language": "es", "voice": "Ana Florence", "streaming": "false"})
open("output.wav", "wb").write(r.content)

# Voice cloning from a reference clip.
with open("reference.wav", "rb") as f:
    r = requests.post("http://localhost:8000/generate",
                      data={"text": "This voice is cloned.", "streaming": "false"},
                      files={"audio": ("reference.wav", f, "audio/wav")})

Text normalization reads numbers with num2words (a dependency). Some languages need optional packages: pypinyin (Chinese), cutlet (Japanese) and ko-speech-tools (Korean). Digits in Chinese text are not converted to words.

Sampling#

The defaults are those of Coqui’s inference: temperature 0.75, top-k 50, top-p 0.85 and a repetition penalty of 10 (the server’s --temperature, --top-k, --top-p and --repetition-penalty override them). Generation stops at the model’s stop code or after 602 codes (about 28 s of audio).

Implementation Notes#

  • The GPT runs in float16 (FlashInfer attention has no float32 kernels, and float16 keeps more precision than bfloat16 over its 30 layers); the final norm and code head, the conditioning encoder, the speaker encoder and the vocoder run in float32, as in Coqui’s inference.

  • The vocoder decodes windows of 24 latents (4 of context, 16 emitted, 4 of look-ahead), aligned to the global mel-frame grid of its 22.05 to 24 kHz resampling and zero-padded after the request’s end, so the streamed audio equals a whole-sequence decode. The first chunk holds 8 latents (about 0.37 s of audio) and is sent after 12 generated codes.

  • Reference clips are encoded when the request is prepared (the 16 most recent clips are cached), on the server’s prompt path: many concurrent requests with new reference clips queue behind that encoding.