agentsclimarketplace
Sign in

Vllm omni audio tts

Skill hsliuustc0106/vllm-omni-skills/skills/vllm-omni-audio-tts

Generate audio and speech with vLLM-Omni using Qwen3-TTS, Fish Speech S2 Pro, CosyVoice3, MiMo-Audio, and Stable-Audio models. Use when synthesizing speech from text, generating audio effects or music, configuring TTS parameters, cloning voices, adding new TTS models, or working with text-to-speech models.From its SKILL.md

Install
npx -y skills add hsliuustc0106/vllm-omni-skills --skill vllm-omni-audio-tts

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

SKILL.md

10.1 KB, as published. Nobody here has run it

vLLM-Omni Audio & TTS

Overview

vLLM-Omni supports text-to-speech (TTS), text-to-audio (sound effects, music), and audio understanding through multiple model families. TTS models use a two-stage autoregressive pipeline (Code Predictor + Code2Wav decoder), while audio generation uses diffusion.

Supported Audio Models

ModelHF IDTypeMin VRAM
Qwen3-TTS 1.7B CustomVoiceQwen/Qwen3-TTS-12Hz-1.7B-CustomVoiceTTS + voice cloning8 GB
Qwen3-TTS 1.7B VoiceDesignQwen/Qwen3-TTS-12Hz-1.7B-VoiceDesignTTS + voice design8 GB
Qwen3-TTS 1.7B BaseQwen/Qwen3-TTS-12Hz-1.7B-BaseBasic TTS8 GB
Qwen3-TTS 0.6B CustomVoiceQwen/Qwen3-TTS-12Hz-0.6B-CustomVoiceTTS + voice cloning4 GB
Qwen3-TTS 0.6B BaseQwen/Qwen3-TTS-12Hz-0.6B-BaseBasic TTS4 GB
Fish Speech S2 Profishaudio/s2-proTTS + voice cloning (dual-AR + DAC)16 GB
CosyVoice3 0.5BFunAudioLLM/Fun-CosyVoice3-0.5B-2512TTS (AR + flow matching)4 GB
MiMo-Audio-7BXiaomiMiMo/MiMo-Audio-7B-InstructAudio understanding + TTS24 GB
MiMo-V2.5-ASRXiaomiMiMo/MiMo-V2.5-ASRASR (speech-to-text)24 GB
OmniVoicenvidia/OmniVoiceTTS + voice cloning (HiggsAudioV2)8 GB
VoxCPM2openbmb/VoxCPM2TTS (native AR, 30+ languages)8 GB
Stable-Audio-Openstabilityai/stable-audio-open-1.0Text-to-audio (music/effects)8 GB

OmniVoice supports voice cloning via ref_audio + ref_text (requires transformers>=5.3). VoxCPM2 is a 2B tokenizer-free native AR TTS model producing 48kHz audio in 30+ languages (requires pip install voxcpm).

Model Architectures

Both Qwen3-TTS and CosyVoice3 use a two-stage autoregressive pipeline. See the reference docs for architecture details, key files, and model variants:

Quick Start: Text-to-Speech

Offline

from vllm_omni.entrypoints.omni import Omni

omni = Omni(model="Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice")
outputs = omni.generate("Hello, welcome to vLLM-Omni!")
audio = outputs[0].request_output[0].audio
audio.save("greeting.wav")

Online API

vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8091

curl -s http://localhost:8091/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
    "input": "Hello, welcome to vLLM-Omni!",
    "voice": "default"
  }' --output greeting.wav

Voice Cloning (CustomVoice variants)

Clone a voice from a reference audio sample:

omni = Omni(model="Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice")
outputs = omni.generate(
    prompt="This is a test of voice cloning with vLLM-Omni.",
    audio_references=["reference_voice.wav"],
)
outputs[0].request_output[0].audio.save("cloned_speech.wav")

Voice Design (VoiceDesign variant)

Design a voice by describing its characteristics:

omni = Omni(model="Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign")
outputs = omni.generate(
    prompt="Welcome to our product launch event!",
    voice_description="A warm, professional female voice with a calm tone",
)
outputs[0].request_output[0].audio.save("designed_voice.wav")

Text-to-Audio (Music & Effects)

Generate music or sound effects with Stable-Audio-Open:

vllm serve stabilityai/stable-audio-open-1.0 --omni --port 8091

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8091/v1", api_key="unused")

response = client.chat.completions.create(
    model="stabilityai/stable-audio-open-1.0",
    messages=[{"role": "user", "content": "Relaxing piano music with rain sounds"}],
)

Audio Understanding (MiMo-Audio)

MiMo-Audio can both understand audio input and generate speech:

omni = Omni(model="XiaomiMiMo/MiMo-Audio-7B-Instruct")

# Transcribe/understand audio
outputs = omni.generate(
    prompt="What is being said in this audio?",
    audio_inputs=["recording.wav"],
)
print(outputs[0].request_output[0].text)

Stage Configuration (Qwen3-TTS)

async_scheduling is enabled by default for Qwen3-TTS models, improving first-packet latency and throughput.

Default stage config uses async_chunk streaming (qwen3_tts.yaml). Key knobs:

ConfigDescriptionDefault
async_chunkEnable inter-stage streamingtrue
runtime.max_batch_sizeMax requests batched per stage1
enforce_eagerDisable CUDA Graph (Stage 0: false, Stage 1: true)varies
codec_chunk_framesAR frames per async chunk (inter-stage streaming only)25
codec_left_context_framesSliding context window for smooth boundaries25
initial_codec_chunk_framesFrames for first emitted codec chunk only (lowers TTFA)0
decode_chunk_framesCode2Wav internal decode chunk size (independent of codec streaming)300
decode_left_context_framesCode2Wav internal left context for decode25

Connector streaming chunking (codec_chunk_frames / codec_left_context_frames) is decoupled from Code2Wav internal decode chunking (decode_chunk_frames / decode_left_context_frames). The connector controls inter-stage streaming windows only, while Code2Wav keeps its own independent decode parameters. Use initial_codec_chunk_frames to emit a small first chunk for low TTFA, then subsequent chunks return to the normal codec_chunk_frames window.

The uniproc Code2Wav stage default max_num_seqs is now 10 (was 1). Avoid reducing below 10 for latency-sensitive deployments.

CUDA Graph warmup for Qwen3-TTS now accounts for custom decode_chunk_frames / decode_left_context_frames overrides.

High-Concurrency Profile

For high-concurrency TTS serving (voice cloning, c=64+), use qwen3_tts_high_concurrency.yaml:

vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni \
  --stage-configs-path vllm_omni/deploy/qwen3_tts_high_concurrency.yaml

This profile enables batched CUDA graph decoder, prefix CUDA graphs for the code predictor, bounded reference-code context, and first-chunk fast emit (initial_codec_chunk_frames: 1). Tuned for 2-GPU serving with Seed-TTS voice-clone workload. Median TTFP is higher than default profile; use for throughput/E2E rather than first-packet-latency optimization.

Additional high-concurrency knobs available in the deploy config:

  • decode_cudagraph_batch_sizes: Multi-batch-size CUDA graph capture for Code2Wav
  • decode_batch_bucket_frames / decode_batch_max_size: Variable-length chunk batching
  • ref_code_context_frames: Limits reference-audio code frames per chunk for stable stage-1 shapes
  • decode_enable_tf32: true: Opt-in TF32 for Code2Wav
  • code_predictor_prefix_graphs: true: Prefix CUDA graph warmup for Stage0 code predictor

For batch mode (no streaming), use qwen3_tts_batch.yaml.

Fish Speech uses fish_speech_s2_pro.yaml with similar knobs. Its DAC codec outputs at 44.1 kHz (vs Qwen3-TTS's 24 kHz).

Note: CosyVoice3 does not support async_chunk streaming yet - use cosyvoice3.yaml (batch mode only).

Streaming Audio

For real-time TTS streaming:

response = client.chat.completions.create(
    model="Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
    messages=[{"role": "user", "content": "A long paragraph of text to stream..."}],
    stream=True,
)

Adding a New TTS Model

For a step-by-step guide on integrating a new TTS model into vLLM-Omni, see the TTS model developer guide. Offline examples are consolidated under examples/offline_inference/text_to_speech/<model>/end2end.py, and online serving examples under examples/online_serving/text_to_speech/<model>/.

Troubleshooting

Audio quality issues: Ensure reference audio for voice cloning is clean (no background noise), 10-20 seconds, single speaker.

Qwen3-TTS code predictor crash: Fixed in #1619. If you encounter a crash in the code predictor stage, update to the latest vllm-omni.

Qwen3-TTS NaN on fp16-only GPUs: The code predictor auto-upcasts to float32 for numerical stability on GPUs without bf16 support (Turing, Volta). No manual override needed. Fixed in #3253.

Qwen3-TTS speaker_embedding dimension error: Speaker embedding dimensions must match the model's talker hidden_size (2048 for 1.7B, 1024 for 0.6B). Mismatched dimensions return HTTP 400. Fixed in #3191.

Qwen3-TTS load_format: dummy: speaker_encoder is always constructed at init time. Voice cloning works under load_format: dummy without extra configuration. Fixed in #3117.

Slow generation: TTS models are autoregressive - generation time scales with output duration. Enable async_chunk for lower first-packet latency. For throughput, increase max_batch_size.

Fish Speech voice cloning latency: Uploaded voices via /v1/audio/voice/upload now auto-cache DAC-encoded reference audio. First request encodes the reference; subsequent requests reuse the cached codes for faster TTFP. Fixed in #2609.

Event loop blocking under concurrent TTS: Blocking tokenizer operations (_build_voxtral_prompt, _build_fish_speech_prompt) now run in a shared ThreadPoolExecutor(max_workers=1). This prevents /health latency spikes under concurrent load. Fixed in #2511.

References

Keep looking

Skills are one crate of 627,801. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.