Essential cookies keep Instavar working. Optional analytics help us understand how the site is used. Cookie Policy
Manage Cookie Preferences
Service reliability telemetry, including Sentry error monitoring and Vercel Speed Insights, stays enabled so we can secure the product and diagnose failures.
Adaptation planner + Failure ownerTurn an audible or training symptom into the next test.
A training run can complete while stale tokens, shape assumptions, voice drift, or weak listening evidence still invalidate the result.
Do this first
Identify the earliest failed contract: corpus, speech tokens, tensor shape, training signal, checkpoint loading, or generated speech.
Keep this boundary in view
The recorded failure and rerun plan belong to the tested corpus and repository state, not every CosyVoice deployment.
Inspect the evidence and what would change the answer
First-party failed run, diagnosis, and corrected rerun path
What supports it
The guide records the full-SFT result, LoRA route, token and shape failures, checkpoints, and remaining listening gate.
Change our mind if
A clean reproduction shows the diagnosed causes were incidental and a simpler unchanged workflow produces stable held-out speech.
Claim TTS-COSYVOICE-DIAG-2026-08 / v1, checked 7 Aug 2026
60-second takeaway Our first CosyVoice3 run used full SFT (all 506M params) and failed after epoch 1 - massive overfitting, 174 GB of checkpoints, grad_norm explosion. The corrected LoRA rerun (2.16M trainable params via PEFT) reached its best checkpoint at epoch 12, with 8.3 MB adapters and stable gradients. This post covers the diagnosis, the fix, 9 known pitfalls, and the companion repo.
If you searched for CosyVoice fine-tuning guide, CosyVoice fine-tuning data requirements, CosyVoice fine-tuning consumer GPU, or CosyVoice voice cloning quality, start here. The important lesson is not that CosyVoice is bad. It is that the training mode decides the outcome: full SFT failed quickly in our run, while the corrected LoRA path produced a stable checkpoint that still needs listening evaluation before production use.
For founders: the LoRA rerun produced a viable checkpoint (epoch 12). Listening evaluation is pending before production deployment.
For engineers: the companion repo has the PEFT-integrated training script, 9 pitfalls, and corrected hyperparameters.
Decision shortcut:
Use this CosyVoice LoRA path when you want a consumer-GPU fine-tuning experiment with small adapter files and explicit checkpoint control.
Use CosyVoice 2 or CosyVoice 3 zero-shot first when you need a quality reference before spending training time.
Use Qwen3-TTS LoRA when you need adapter scale control at inference time.
Use VoxCPM 1.5 when you need the lowest-friction fine-tuned voice that already passed our production listening checks.
LoRA rerun results
The corrected LoRA run (PEFT, 2.16M trainable params, LR 5e-5) on IMDA NSC FEMALE_01 (16,535 train / 870 dev utterances):
Metric
LoRA rerun
Failed full-SFT run
Trainable params
2.16M (0.44%)
506M (100%)
Best CV loss
3.044 (epoch 12)
2.900 (epoch 1)
Epochs before overfit
12
1
Checkpoint size
8.3 MB
4 GB
Grad norm (best region)
1.4-4.0
4.1-19+
Peak VRAM
6.96 GB
13.08 GB
Training speed
6.9 samples/sec
3.8 samples/sec
Listening evaluation of the epoch 12 checkpoint is pending. The LoRA approach produced a much more stable training run with 12 epochs of genuine improvement before divergence.
What went wrong in the first run
CosyVoice2 (baseline/control)
CosyVoice3 (first run - full SFT)
Training mode
No finetune - zero-shot reference
Full SFT (all 506M params trained)
Qualitative result
Acceptable naturalness, usable as control
Did not reach production quality
Long-form stability
Stable
Unstable beyond ~20 seconds
Linguistic consistency
Consistent
Weak in listening checks
The critical discovery: the first run used full SFT via the upstream training pipeline, not LoRA. The train_cosyvoice3_lora.py script existed but was not the training path that produced the evaluated checkpoints. This caused catastrophic forgetting of the pretrained model's generalization.
Audio evidence
CosyVoice2 baseline/control
CosyVoice3 representative sample (this run - not production-ready)
Listen to the two clips side by side. The CosyVoice2 clip is cleaner on naturalistic prosody. The CosyVoice3 clip shows the instability in long-form decoding that blocked production deployment.
Failure mode analysis
Three contributing factors explain the current run outcome:
1. Checkpoint quality drift
CosyVoice3 LoRA training showed a pattern where early epochs were noticeably better than later ones. We did not have a strict per-epoch validation gate in place, so the run continued past the best region. The practical lesson: CosyVoice3 needs tighter checkpoint gating with explicit listening checkpoints every 2-3 epochs, not just loss curve tracking.
2. Prompt sensitivity and long-text decoding
CosyVoice3 in this run was sensitive to prompt formatting. Small changes to how the text was segmented before inference changed output quality significantly. Long-text generation (>20s) showed linguistic inconsistency - words dropped or merged in a way that sounded unnatural on listening review. VoxCPM and Qwen3-TTS were more robust to prompt formatting variation.
3. Operational fragility: large checkpoint churn
The training loop produced large checkpoint files at short intervals. Without explicit retention policy, older (potentially better) checkpoints were overwritten. By the time we evaluated, some of the best early-epoch checkpoints were no longer available. This is the same issue that affected the IndexTTS2 run, but with a more acute impact here because the best zone was earlier.
Streaming and long-form reality check
CosyVoice is attractive when a demo clip sounds natural, but production use depends on whether the same voice survives streaming, prompt variation, and longer text. Treat CosyVoice as a stronger candidate for pre-produced narration than for live voice agents until your own first-audio latency and chunk-stability checks pass.
Failure mode
What users notice
What to test before shipping
First-chunk latency
The assistant feels slow even if the full clip renders quickly
Measure time to first audible audio, not only full-file real-time factor
Stutter
Short repeats or broken rhythm during streaming
Test the same text in batch and streaming paths, then compare artifacts
Voice drift
The speaker changes across chunks or segments
Reuse the same reference handling and compare multi-paragraph outputs
Swallowed words
Words disappear or merge in longer prompts
Keep a held-out script with names, numbers, punctuation, and short clauses
Prompt leakage
Formatting or reference text affects the spoken output
Test clean text, markdown-like text, and segmented text separately
Long-text collapse
Quality drops after the first 20 to 30 seconds
Evaluate the actual passage length you plan to publish, not only a one-sentence sample
Checkpoint churn
A later checkpoint sounds worse even when loss looks better
Preserve checkpoints and run listening checks every few epochs
The practical split is simple: use CosyVoice when you can batch-generate, review, cut, and regenerate audio before publishing. Prefer Qwen3-TTS for latency-first voice agents, VoxCPM 1.5 for the lowest-friction fine-tuned production path, or the TTS model decision tree when licensing, hardware, or voice-cloning quality is the first constraint.
What the LoRA rerun fixed
The corrected LoRA run addressed all three failure modes:
1. LoRA instead of full SFT
└─ 2.16M trainable params (0.44%) via PEFT
└─ 8.3 MB checkpoints (480x smaller)
└─ No catastrophic forgetting - base model frozen
2. Stable training curve
└─ 12 epochs of genuine improvement (vs 1 for full SFT)
└─ Grad norm stayed at 1.4-4.0 (no explosion)
└─ CV loss bottomed at 3.044 (epoch 12)
3. All checkpoints preserved
└─ 200 checkpoints at 8.3 MB each = 1.7 GB total
└─ Best epoch (12) available for evaluation
└─ Compare: old run had 43 checkpoints at 4 GB each = 174 GB
Is CosyVoice LoRA fine-tuning worth trying on a 24GB GPU?
Yes. The LoRA rerun peaked at 6.96 GB VRAM on an RTX 3090 Ti. The companion repo (instavar/cosyvoice3-lora-finetuning) has the PEFT-integrated training script and 9 documented pitfalls.
Why did the first run fail?
The first run used full SFT (all 506M parameters), not LoRA. This caused catastrophic forgetting after epoch 1. The corrected LoRA rerun (2.16M params, LR 5e-5) trained stably for 12 epochs.
What learning rate should I use?
5e-5 for LoRA (not 1e-5, which was used for full SFT). The first run's artifact path female01_cv3_lr1e5_run1 reflects the full SFT LR. The LoRA rerun used female01_cv3_lora_lr5e5_run1.