Essential cookies keep Instavar working. Optional analytics help us understand how the site is used. Cookie Policy
Manage Cookie Preferences
Service reliability telemetry, including Sentry error monitoring and Vercel Speed Insights, stays enabled so we can secure the product and diagnose failures.
Named-model investigator + Choice-makerCompress the field into models worth a closer look.
Codec, training objective, adaptation support, sample rate, and licence determine whether two TTS models can share a workflow.
Do this first
Compare the data pipeline and adaptation boundary before comparing parameter counts or demo quality.
Keep this boundary in view
Architecture explains constraints and likely failure modes; it does not predict listener preference by itself.
Inspect the evidence and what would change the answer
Architecture audit plus selected adaptation runs
What supports it
The comparison traces six model families through tokenization, training, data preparation, licensing, and deployment implications.
Change our mind if
A common workflow reproduces equivalent adaptation behavior across these architectures without hiding model-specific constraints.
Claim TTS-ARCHITECTURE-2026-08 / v1, checked 7 Aug 2026
60-second takeaway These six open-source TTS models look similar on paper, but their architectures lead to different preprocessing, adaptation, evaluation, and licensing constraints. By 19 July 2026, we had exercised real adaptation paths on all six, ranging from production-selected full SFT to bounded LoRA pilots. The evidence is not equally strong: Qwen3-TTS and IndexTTS2 have selected checkpoints, while Voxtral, Fish Speech, Chatterbox, and CosyVoice3 still have important quality boundaries. This article explains the architecture before you commit GPU hours.
Who this is for
Engineers choosing a TTS model to fine-tune: You have read the benchmarks. You know which models exist. You need to understand the architectural differences before committing to a fine-tuning approach - because picking the wrong approach wastes days, not hours.
ML engineers adding TTS to a pipeline: You need to know which models wrap with standard PEFT, which need custom LoRA libraries, and which only support full SFT - before you design your training infrastructure.
Technical leads evaluating licenses: Two of these six models have license restrictions that are not obvious from their GitHub repos. This article flags them before you build on top of them.
The six models
Model
Total params
LLM backbone
Released
License
Voxtral 4B
~4.1B
Ministral-3B
March 2026
CC BY-NC 4.0
Qwen3-TTS 1.7B
1.7B
Qwen3
January 2026
Apache-2.0
IndexTTS2
~560M
Custom GPT-2
June 2025
Custom Bilibili Model Use License
Chatterbox
500M
Llama 3-inspired
June 2025
MIT
Fish Speech S2 Pro
~5B
Qwen3-4B
March 2026
Fish Audio Research License
CosyVoice 3
~800M
Qwen2-0.5B
May 2025
Apache-2.0
These are the six models selected for the original March 2026 architecture comparison. Models where upstream already provided complete fine-tuning tooling, such as VoxCPM 1.5 and GPT-SoVITS v4, were excluded from the architecture table. They still contribute useful evidence to our broader adaptation laboratory.
Three generation paradigms
Every model in this list generates speech through a multi-stage pipeline. The stages differ, and that difference determines your fine-tuning strategy. Three paradigms cover all six models.
Text → LLM predicts speech tokens (autoregressive)
→ Flow-matching model converts tokens to mel/acoustic representation
→ Vocoder synthesizes waveform
The LLM backbone is the primary fine-tuning target. The flow-matching decoder and vocoder are typically frozen - they handle acoustic rendering, not linguistic content or speaker identity.
Why this matters for fine-tuning: You only need to LoRA or SFT the backbone. The flow-matching decoder (100–390M parameters depending on the model) and vocoder are left untouched. This makes fine-tuning cheaper than the total parameter count suggests - Voxtral is 4.1B total, but you only fine-tune the 3.4B backbone.
Voxtral specifics: The backbone predicts one semantic token per frame from an 8,192-entry codebook. The 390M flow-matching transformer then runs 8 Euler steps per frame to produce 36 acoustic values. The paper explicitly states: freeze text-embedding layers during fine-tuning to prevent overfitting on rare vocabulary. The 300M Voxtral Codec is both the tokenizer and the vocoder - it encodes reference audio into tokens and decodes generated tokens back to 24 kHz waveforms.
Chatterbox specifics: The T3 component (text-to-speech-token, Llama-based) is the fine-tuning target. S3Gen (flow-matching) and the HiFT-GAN vocoder are frozen. The S3Tokenizer is shared with CosyVoice 2 - data prepared for one codec family partially transfers.
CosyVoice 3 specifics: The Qwen2 backbone predicts speech tokens from a 6,561-entry vocabulary. The DiT flow-matching decoder depth=22,dim=1024,300Mparams converts tokens to 80-band mel spectrograms. A causal HiFiGAN variant produces 24 kHz audio.
Paradigm B: Multi-codebook autoregressive → codec decoder (no separate flow stage)
Models: Qwen3-TTS, Fish Speech S2 Pro
Text → AR model predicts all codebook layers (semantic + residual)
→ Codec decoder reconstructs waveform directly from tokens
There is no separate flow-matching or diffusion stage. The autoregressive model handles both semantic and acoustic token prediction. A lightweight codec decoder (not a full vocoder) converts tokens to audio.
Why this matters for fine-tuning: One LoRA pass covers the entire generation pipeline. You do not need to decide whether to also fine-tune a separate flow model. But the model is doing more work per forward pass (predicting multiple codebook layers), which affects training dynamics.
Qwen3-TTS specifics: A dual-track architecture - the talker predicts codebook-0 (semantic layer) autoregressively, then an MTP (Multi-Token Prediction) sub-talker predicts 15 residual codebooks simultaneously at each position. Total: 16 codebook layers at 12.5 Hz. The model registers as Qwen3TTSForConditionalGeneration - standard HuggingFace PEFT wrapping works natively. This is the simplest LoRA integration of any model in this list.
Fish Speech S2 Pro specifics: DualAR architecture - a 4B Slow AR (built on Qwen3-4B) predicts codebook-0 at each timestep, then a 400M Fast AR (4-layer shallow transformer) predicts codebooks 1–9 depth-wise. The Slow AR hidden state conditions the Fast AR. This is structurally isomorphic to a standard causal LM, so it inherits LLM serving optimizations (KV-cache, speculative decoding). However, LoRA uses a custom loralib implementation, not HuggingFace PEFT.
Paradigm C: Autoregressive backbone → DiT mel predictor → vocoder
Models: IndexTTS2
Text → GPT predicts semantic codes (autoregressive)
→ DiT converts semantic codes to mel spectrogram (non-autoregressive)
→ BigVGAN synthesizes waveform
The GPT backbone is the fine-tuning target. The S2M DiT (Semantic-to-Mel Diffusion Transformer) and BigVGAN vocoder are frozen.
Why this matters for fine-tuning: This is the only model that predicts semantic codes and then uses a separate non-autoregressive model to produce mel spectrograms. The DiT adds quality but also adds a stage that cannot be LoRA'd with standard tools - the GPT backbone is a custom GPT-2, not a HuggingFace PreTrainedModel. Full SFT is the only fine-tuning path.
IndexTTS2 specifics: The GPT backbone is ~560M parameters (24 layers, dim 1280, 20 heads) - significantly smaller than often cited. It uses a MaskGCT semantic codec (SeamlessM4T → Wav2Vec2Bert → RepCodec) with an 8,192-entry codebook. Training loss is weighted: 0.2 text cross-entropy + 0.8 mel cross-entropy.
Paradigm summary
Paradigm
Models
Fine-tuning target
What's frozen
LoRA possible?
A: AR + flow-matching + vocoder
Voxtral, Chatterbox, CosyVoice 3
LLM backbone
Flow model + vocoder
Yes (backbone only)
B: Multi-codebook AR + codec
Qwen3-TTS, Fish Speech S2 Pro
Full AR model
Codec encoder/decoder
Yes (single LoRA pass)
C: AR + DiT + vocoder
IndexTTS2
GPT backbone
DiT + BigVGAN
No (full SFT only)
Codec and tokenizer comparison
The codec determines what the model is actually learning to predict - and misunderstanding this is the fastest way to waste a fine-tuning run.
Model
Codec
Token rate
Semantic vocab
Tokens per frame
Output sample rate
Voxtral 4B
Voxtral Codec (VQ + FSQ hybrid)
12.5 Hz
8,192
37 (1 semantic + 36 acoustic)
24 kHz
Qwen3-TTS
Custom RVQ (16 codebooks)
12.5 Hz
2,048 per layer
16
24 kHz
IndexTTS2
MaskGCT semantic (RepCodec)
~25 Hz
8,192
1
22 kHz
Chatterbox
S3Tokenizer (FSQ)
25 Hz
6,561
1
24 kHz
Fish Speech S2 Pro
Modified DAC (RVQ)
~21 Hz
4,096
10 (1 slow + 9 fast)
44.1 kHz
CosyVoice 3
FSQ-MinMo (multi-task)
25 Hz
6,561
1
24 kHz
What the token rate tells you
Lower token rate (12.5 Hz) means fewer tokens per second of audio, which means shorter training sequences and faster training per sample. Higher token rate (25 Hz) means more temporal resolution but longer sequences.
Voxtral and Qwen3-TTS at 12.5 Hz produce half as many tokens per second as IndexTTS2 or Chatterbox at 25 Hz. For a 10-second audio clip, that is 125 tokens vs 250 tokens - a 2x difference in sequence length that directly affects training memory and speed.
What the tokens-per-frame count tells you
Voxtral predicts 37 tokens per frame (1 semantic + 36 acoustic). Qwen3-TTS predicts 16 tokens per frame. Chatterbox and CosyVoice 3 predict just 1 token per frame.
More tokens per frame means more information is being generated per timestep, which generally means higher audio quality but also more complex training dynamics. Models with 1 token per frame rely on a separate flow-matching or DiT stage to reconstruct the acoustic details.
The sample rate trap
Five of six models operate at 22–24 kHz. Fish Speech S2 Pro operates at 44.1 kHz. If you switch between models without checking sample rate, training may appear to work - loss decreases normally - but output quality degrades silently. This was the #1 silent failure across our fine-tuning runs. Always verify sample rate before every training run:
# Check before training - every time
ffprobe -v error -show_entries stream=sample_rate -of csv=p=0 your_audio.wav
# Resample if needed
ffmpeg -i input.wav -ar 24000 output_24k.wav # For most models
ffmpeg -i input.wav -ar 44100 output_44k.wav # For Fish Speech
Codec family relationships
Chatterbox's S3Tokenizer is derived from the same family as CosyVoice 2's S3TokenizerV2. CosyVoice 3 upgraded to a new FSQ-MinMo tokenizer trained on multi-task objectives (ASR, speech emotion recognition, language ID, audio event detection). This means CosyVoice 3 tokens carry richer information than CosyVoice 2 tokens - prosody, emotion, and speaker style are encoded, not just phonemic content.
LoRA compatibility: why some models wrap with PEFT and others don't
Not all LoRA is the same. The architecture of the backbone determines which LoRA framework works - and getting this wrong means your training script crashes before the first step.
Model
LoRA framework
Why
PEFT get_peft_model() works?
Qwen3-TTS
HuggingFace PEFT
Talker is a standard Qwen3 transformer decoder
Yes - native
Voxtral 4B
HuggingFace PEFT (expected)
Ministral backbone is HF-compatible
Likely yes - untested
CosyVoice 3
HuggingFace PEFT (on Qwen2)
Qwen2 backbone is HF-compatible
Yes for backbone, not for DiT
Chatterbox
HuggingFace PEFT or custom
Llama-based T3 component
Partial - community repos exist
Fish Speech S2 Pro
Custom loralib
Not PEFT - uses setup_lora() that replaces nn.Linear layers
No
IndexTTS2
None
Custom GPT-2, not a PreTrainedModel
No - full SFT only
The PEFT compatibility rule
If the backbone inherits from a HuggingFace PreTrainedModel (Qwen2, Qwen3, Llama, Ministral), standard PEFT LoRA wrapping works. If the backbone is custom (IndexTTS2's GPT-2, Fish Speech's DualAR), you either need the upstream's custom LoRA implementation or you are limited to full SFT.
This is an architectural constraint, not a library limitation. Fish Speech's DualAR uses loralib (a separate library from PEFT) with a custom setup_lora() function that walks the model and replaces nn.Linear layers. You cannot swap in PEFT without rewriting the injection logic.
LoRA target modules
For models where LoRA works, the target modules follow the backbone architecture:
qkv_proj, o_proj, gate_up_proj, down_proj (from Voxtral Mini ASR)
32
32
Fish Speech S2 Pro
attention, mlp, embeddings, output (both Slow + Fast AR)
8
16
CosyVoice 3
Qwen2 attention + MLP layers
TBD
TBD
Qwen3-TTS and Voxtral target the standard attention projections and MLP gate/up/down projections - the same pattern as any Qwen or Mistral language model. Fish Speech targets broader module categories because its custom loralib uses string matching on module names rather than explicit layer references.
Data pipeline complexity
This is where the models diverge most dramatically. Preprocessing time ranges from minutes to over 30 minutes for the same dataset - and skipping a step does not produce an error, it produces bad output.
What the preprocessing tells you about iteration speed
If you are running multiple experiments, preprocessing time compounds. The bounded Chatterbox run reached 512 optimizer steps in about 2 minutes 20 seconds, while the recorded IndexTTS2 preprocessing path took more than 30 minutes. Those measurements show different iteration costs, but they are not equivalent end-to-end training runs and do not by themselves make Chatterbox the best exploration model.
Data format summary
Model
Input
Training format
Chatterbox
wav (16 kHz auto-resampled) + txt
LJSpeech format or wav+txt pairs
CosyVoice 3
wav (24 kHz) + text + instruct file
Parquet (1000 utterances per file)
Qwen3-TTS
wav (24 kHz) + transcript
JSONL with pre-extracted 16-layer codec codes
Voxtral 4B
wav (24 kHz) + transcript
Local reconstructed token and prompt artifacts; no stable public prep contract yet
Fish Speech S2 Pro
wav (44.1 kHz) + .lab transcript
Protobuf dataset
IndexTTS2
wav (24 kHz resampled) + transcript
JSONL manifest + .npy feature files
CosyVoice 3 uniquely requires an instruct file in addition to standard fields - this is new vs CosyVoice 2 and is a common source of configuration errors when adapting CosyVoice 2 recipes.
What we learned fine-tuning on the same corpus
We exercised adaptation paths on all six models using the IMDA NSC FEMALE_01 corpus slice and an RTX 3090 Ti (24GB). The official dataset structure identifies FEMALE_01 as a TTS voice, but that label alone should not be treated as independent proof of the number or identity of people represented in every local artifact.
Model
Fine-tuning type
Steps/epochs completed
Best checkpoint
Production-ready?
Qwen3-TTS
LoRA
17 epochs (2,950 steps)
Epoch 10
Yes
IndexTTS2
Full SFT
15,949 steps
Step 14,000
Yes
CosyVoice 3
LoRA (custom scripts)
Run completed
Epoch 12 by CV loss
Listening pending
Chatterbox
T3 full SFT
512 steps (2 min 20s)
Quality eval pending
Trained, unreviewed
Fish Speech S2 Pro
LoRA plus full-SFT feasibility
64-step LoRA; 10-step full-SFT maximum
Pilot only
Feasibility only
Voxtral 4B
Prompt adaptation plus LoRA
128-example, 5-epoch LoRA comparison
Did not beat prompt baseline
Not promoted
Three patterns that recurred across the checked runs
1. The best checkpoint was often not the last one.
VoxCPM peaked at step 4,000 (not 8,999). IndexTTS2 peaked at step 14,000 (not 15,949). Qwen3-TTS peaked at epoch 10 (not 17). These runs justify keeping checkpoints and evaluating them on held-out prompts. They do not prove that the final checkpoint can never be best in another run.
2. Sample rate mismatches are the #1 silent failure.
Qwen3-TTS requires 24 kHz. Fish Speech requires 44.1 kHz. IndexTTS2 resamples input to 24 kHz but outputs at 22 kHz. If you switch between models without checking, training appears to work (loss decreases normally) but output quality degrades. We hit this on three separate occasions before making sample rate verification the first step of every run.
3. Default hyperparameters are wrong for single-speaker fine-tuning.
In our checked Qwen3-TTS run, LR 2e-5 produced noise and 2e-6 worked better. Adapter scale 1.0 over-steered that voice, while 0.3 to 0.35 was selected. Treat these as reproducible starting evidence for that configuration, not universal constants. Run a bounded sweep before committing a longer job.
Model-specific findings
Qwen3-TTS had a double label-shift bug in the upstream training script (sft_12hz.py) - the script manually shifts inputs/labels before calling the model, but ForCausalLMLoss in HuggingFace Transformers also shifts internally. This causes the model to learn predictions two positions ahead, which manifests as progressively faster speech with each training epoch. We fixed this in our companion repo and submitted upstream PR #178.
IndexTTS2 required explicit checkpoint retention management - the default policy deletes older checkpoints before you can evaluate them. The best checkpoint (step 14,000) was not the final step (15,949). We wrote and open-sourced the entire fine-tuning pipeline since the official repo provides inference only: instavar/indextts2-finetuning.
CosyVoice 3 had upstream configuration bugs (GitHub issue #1680) - the speech_token_size in the yaml was stale from CosyVoice 2 (6564 vs 6561), causing shape mismatches during LLM training. Our first run also showed checkpoint drift and long-text instability, which we attribute to training only the LLM while keeping the flow model frozen - the LLM learns new token distributions that the frozen DiT has not seen.
Chatterbox completed 512 optimizer steps in 2 minutes 20 seconds. The log reports about 0.0425 epoch, so this was not a 512-sample dataset claim. The run proves fast bounded training and checkpoint production; unique-example coverage, comparative listening quality, and any minimum viable dataset threshold remain unverified.
Fish Speech S2 Pro required careful protobuf version management - the DAC dependency needs protobuf<5, which conflicts with most modern ML environments. VQ extraction for ~7 hours of audio took 22 minutes, making it the second-slowest preprocessing pipeline after IndexTTS2.
License matrix: what you can actually deploy
Several of these models have terms or deployment conditions that are easy to miss. Code, weights, datasets, and generated output can each carry separate obligations.
Model
Weights license
Commercial fine-tuned weights?
Fine-tuning code
Qwen3-TTS
Apache-2.0
Yes
Apache-2.0
IndexTTS2
Custom Bilibili Model Use License
Conditional - review current terms
Apache-2.0 companion code
Chatterbox
MIT
Yes
Apache-2.0
CosyVoice 3
Apache-2.0
Yes
Apache-2.0
Voxtral 4B
CC BY-NC 4.0
No - requires separate Mistral commercial license
Apache-2.0 (ours)
Fish Speech S2 Pro
Fish Audio Research License
No - requires separate Fish Audio commercial license
Apache-2.0 (ours)
If you need a permissive model license in this checked set, start by reviewing Qwen3-TTS, Chatterbox, or CosyVoice 3. IndexTTS2 is a separate custom-license case and should not be described as fully permissive. Recheck the official license for every model before commercial or redistributable use.
If you are doing research or non-commercial work, review the current terms anyway. Research availability does not mean absence of attribution, acceptable-use, redistribution, jurisdiction, or downstream conditions.
The distinction matters because companion fine-tuning code can use Apache-2.0 while model weights retain separate upstream terms. Treat the matrix as engineering routing, not legal advice, and re-attest the official sources before training, redistribution, or deployment.