Audio tts

Audio TTS — local voice-clone text-to-speech

Turn a text file into narrated WAV or MP3 in a chosen (cloned) voice, running entirely on this Mac. A default voice (claude) ships with the skill; add your own by dropping a short reference clip into assets/voices/. Optionally emit word + expression-cue timestamps via forced alignment.

Paths below are relative to this skill directory. If you're not cd'd into it, prefix with the skill's absolute path (e.g. .../skills/audio-tts/scripts/tts.sh).

Platform

Synthesis is Apple-Silicon only — it runs on mlx-speech (Apple's MLX/Metal framework) and does not work on Linux/Windows. If you're not on an Apple-Silicon Mac, use the upstream fish-speech workflow documented in the repo README (§ Linux / Windows); the base model is the same Fish Audio family, so voices carry over. The rest of this skill assumes Apple Silicon.

One-time setup

cd skills/audio-tts
uv venv --python 3.13 .venv                 # or: python3.13 -m venv .venv
uv pip install --python .venv -r requirements.txt   # or: .venv/bin/pip install -r requirements.txt

The launcher scripts/tts.sh auto-uses .venv when present. First run downloads the fish-s2-pro weights (~6 GB) to the Hugging Face cache.

For --timestamps (optional), also create the alignment venv (torch is deliberately kept out of the synthesis venv):

uv venv .venv-align
uv pip install --python .venv-align -r requirements-align.txt

Synthesize

# Default 'claude' voice → MP3 (format chosen by the -o extension)
scripts/tts.sh input.txt -o out.mp3

# A different bundled/added voice
scripts/tts.sh input.txt -o out.wav --voice morgan

# One-off clone from an explicit reference (wav + its transcript)
scripts/tts.sh input.txt -o out.mp3 --reference-audio ref.wav --reference-text "the transcript"

# Model's base voice, no cloning
scripts/tts.sh input.txt -o out.mp3 --no-clone

# Also emit word + expression-cue timestamps (needs the align venv) → out.json
scripts/tts.sh input.txt -o out.mp3 --timestamps

# Plan only — show the chunking, no model load
scripts/tts.sh input.txt -o out.mp3 --dry-run

Input can be a UTF-8 file or - for stdin. Useful flags: --max-chars (chunk size, default 400 ≈ 18 s), --gap-ms (silence between chunks), --bitrate, --normalize-peak, -v (per-stage timings). Run scripts/tts.sh -h for the full list.

The synthesizer splits text into sentence-aligned chunks that stay inside the model's coherence window (≈ a paragraph), clones the reference voice once, synthesizes each chunk, stitches them with a short gap, and peak-normalizes. Generation is ≈ 2–3× realtime.

Adding a voice

A voice is a pair in assets/voices/: <name>.wav + <name>.txt.

• <name>.wav: ~10–30 s of clean, single-speaker audio (the timbre to clone).
• <name>.txt: the exact transcript of that clip.

Then synthesize with --voice <name>. Point at a different folder with --voices-dir.

Expression & pacing

Inline [bracket] tags restyle the following words. Expression cues that render as audio events — [laughing], [sigh], [gasp], [cough], … — are the ones --timestamps locates; other tags (emotion/voice/volume/pacing) only recolor the words and are stripped from the word stream. In practice punctuation out-performs tags for pacing (. ! ?, two spaces after a sentence ender, a blank line for the longest pause). For emphasis, wrap words in [loud] … [normal] rather than ALLCAPS (cleaner, and avoids token-mangling). See references/rendering-notes.md for the full pacing guide and hard-won QA gotchas.

Always audition the render

fish-s2-pro is excellent but occasionally mangles a token (certain ALLCAPS/acronym words can hard-break a render, and it varies per voice). So listen to / verify every render — don't ship blind:

1. Verify with the audio-lens skill (sibling in this repo): check for silence, clipping, and a sane spectrogram. node ../audio-lens/bin/audio-lens.js analyze <out.wav>.
2. If a word came out wrong, tweak just that token (lowercase → phonetic respell "sock two" → letterize S-O-C → reword) and re-render. Keep the working .txt beside the render.

Details and the token-breakage catalog are in references/rendering-notes.md.

Timestamps output

--timestamps writes a JSON sidecar: {audio, sample_rate, duration_ms, text, items:[{type:"word"|"expression", value, s, e}]} (ms). Words come from torchaudio MMS_FA forced alignment; expression cues are placed in the inter-word gap where the model rendered them, refined with Silero VAD. Run scripts/tts-align.sh -h to align an existing audio+text pair directly.