waxal-benchmarking/voxcpm2-multilang10-full

WAXAL VoxCPM2 — 10-Language African TTS

A full fine-tune of VoxCPM2 (2B params, tokenizer-free, diffusion-AR TTS with voice cloning) on the 10 highest-resource languages in the WAXAL project's AfroNet collection, produced by LyngualLabs.

This is a research checkpoint. It is not production-polished — see Known Limitations below before using it for anything beyond experimentation.

A broader, independent 28-language sibling model covering more (generally lower-resource) languages also exists: waxal-benchmarking/voxcpm2-multilang28-full.

Languages

VoxCPM2 has no built-in mechanism for specifying language at inference — the base model infers pronunciation purely from text orthography. We added one: a dedicated <lang:XX> tag per language, registered as a real token in the tokenizer and resized into the text embedding table (mean-initialized, then learned during fine-tuning). The tag is mandatory — always prepend it to the text you want synthesized, e.g.:

<lang:yo> Ẹ káàárọ̀! Báwo ni oorun rí lálẹ́ àná?

Language	Tag	Raw hours	Hours after resampling
Yoruba	<lang:yo>	1038.7	816.3
Swahili	<lang:sw>	1004.1	802.6
Kinyarwanda	<lang:rw>	989.9	796.9
Igbo	<lang:ig>	930.2	772.5
Hausa	<lang:ha>	682.6	661.8
Wolaytta	<lang:wal>	196.6	355.2
Dholuo	<lang:luo>	191.5	350.5
Amharic	<lang:am>	189.2	348.4
Oromo	<lang:om>	189.2	348.4
Sidama	<lang:sid>	188.2	347.5

Source data: the AfroNet speech collection (Professor/<language>-speech-data).

Training

• Base: openbmb/VoxCPM2, with 10 new <lang:XX> tokens added and the text embedding table resized (mean-initialized rows) before any fine-tuning started.
• Mode: full fine-tuning (all parameters trainable except the frozen AudioVAE codec).
• Data balancing: the 10 languages range 5.5x in size (1038.7h Yoruba vs. 188.2h Sidama). We used sqrt-temperature resampling (weight ∝ √hours, renormalized to preserve total volume) to counter this — downsampling the biggest languages to ~79–97% of raw hours and upsampling the smallest to ~1.8x, via a mix of subsampling and controlled repetition.
• Scale: 2,724,675 training rows after resampling, 4,950 held-out validation rows.
• Steps: 42,000 steps (~0.99 epoch over the resampled training set), batch size 3 × grad-accum 11 × 2 GPUs (effective batch 66).
• Hardware: 2× NVIDIA H200, via torchrun DDP.
• Learning rate: 1e-5, 100 warmup steps.
• One real incident worth documenting: at step 15,000 (~36% through), training hit a CUDA OOM on one GPU, caused by memory fragmentation building up over a long run combined with running close to the memory ceiling from the start. The checkpoint at step 15,000 had just saved cleanly, so no progress was lost. We resumed with a smaller per-GPU batch (4→3, compensating grad-accum 8→11 to keep the same effective global batch), a tighter max-batch-token cap (8192→6144) to blunt outlier-length batches, and PyTorch's expandable_segments allocator mode to fight fragmentation directly. The run then completed the remaining 27,000 steps with zero further issues. (This is why there are two W&B runs for this model's training — voxcpm2-multilang10-full for steps 0–15,000, voxcpm2-multilang10-full-resumed for 15,000–42,000 — rather than one continuous run.)

Usage

from huggingface_hub import snapshot_download
from voxcpm import VoxCPM
import soundfile as sf

model_dir = snapshot_download("waxal-benchmarking/voxcpm2-multilang10-full")
model = VoxCPM.from_pretrained(model_dir, load_denoiser=False, optimize=False)

wav = model.generate(
    text="<lang:yo> Ẹ káàárọ̀! Báwo ni oorun rí lálẹ́ àná?",
    cfg_value=2.0,
    inference_timesteps=22,
)
sf.write("out.wav", wav, model.tts_model.sample_rate)

optimize=False is required — torch.compile (the default optimization path) conflicts with some inference environments (e.g. HF ZeroGPU Spaces); disabling it costs some speed but avoids crashes.

Voice cloning works the same way as base VoxCPM2 — pass prompt_wav_path and prompt_text (the literal transcript of what's spoken in the reference clip) alongside the tagged text.

Try it live: WAXAL VoxCPM Multilingual Demo Space.

Evaluation

Unlike the 28-language sibling model, this checkpoint has not yet been run through our automated evaluation pipeline (ASR-based CER via Meta's Omnilingual ASR, with a real-recording noise floor per language, plus DNSMOS naturalness scoring). That pipeline exists and is validated — see the 28-lang model card for methodology and results — it just hasn't been pointed at this specific checkpoint yet. Treat any quality impressions of this model as informal/listening-based until that's done.

Known Limitations

• Voice-design and cross-lingual flexibility are likely reduced. Full fine-tuning on read-speech-only data (even across 10 languages) degrades the base model's text-prompted style/voice-design control and non-trained-language generation, based on findings from earlier single-language full fine-tunes on this same base model. We have not formally verified how much voice-design capability survives in this specific checkpoint.
• No automated evaluation yet for this checkpoint specifically (see above).
• Translation/transcript quality varies by language in our own test tooling (the demo Space's example sentences) — Wolaytta and Sidama in particular had machine-translated examples that looked suspect on inspection. This is a caveat about our test sentences, not necessarily the model.
• The step-15,000 OOM incident (see Training) is disclosed for transparency; the resumed run completed cleanly and we have no evidence it affected final quality, but it hasn't been specifically ruled out either.

Related

• waxal-benchmarking/voxcpm2-multilang28-full — an independent fine-tune on all 28 AfroNet languages, trained the same way, with a validated automated evaluation pipeline already run.
• AfroNet speech collection — the source training data.