brijsri's picture
Update quick start for the new repo id
e02da9c verified
|
Raw History Blame Contribute Delete
4.59 kB
---
license: cc-by-nc-4.0
language:
- en
- fr
library_name: pytorch
pipeline_tag: audio-to-audio
tags:
- speaker-anonymization
- voice-privacy
- speech-editing
- residual-vector-quantization
- bigvgan
- interspeech2026
---
# Acoustic Token Admixture: model weights and speaker pool
Pretrained checkpoints and the pseudospeaker pool for the Interspeech 2026 paper
**"Acoustic token admixture for joint speaker and content anonymization"**
(Ali Golmakani, Seyed Ahmad Hosseini, Omar Manil Bendali, Emmanuel Vincent, Brij Mohan Lal Srivastava).
- **Code:** <https://github.com/Nijta/acoustic-token-admixture>
- **Audio samples:** <https://nijta.github.io/acoustic-token-admixture/>
The system anonymizes both the speaker's voice and named entities in the
content. It mixes encoder-derived and phoneme-conditioned RVQ tokens frame by
frame, then resynthesizes the result with a BigVGAN vocoder conditioned on a
pseudospeaker. On VoicePrivacy 2024 it reaches EER 42.54% with WER 3.73%
(β = 0.7).
## Quick start
```bash
git clone https://github.com/Nijta/acoustic-token-admixture.git
cd acoustic-token-admixture
pip install -r requirements.txt
hf download nijta/acoustic-token-admixture --local-dir models
(cd models && sha256sum -c SHA256SUMS)
export MODELS_DIR=$PWD/models
export PYTHONPATH=".:./audiolm:./bigvgan/src"
python static_infer.py --inputs input.wav --output-dir outputs --seeds 52 \
--gender any --admixture-ratio 0.7 --pitch-f 0.75 --db 2
```
The folder layout matches what the code expects in `$MODELS_DIR`.
## Files
| Path | Component | Paper |
|---|---|---|
| `RVQWhisper/rvq_model_en.pth`, `config.yaml` | 8 x 1024 residual vector quantizer on frozen Whisper large-v2 encoder features (Stream A) | Sec. 2.1.1 |
| `Aligner/aligner_english.pth` | CTC phoneme head (111 classes) for forced alignment | Sec. 2.1.2 |
| `Aligner/dur_predictor.pt`, `f0_predictor.pth` | Duration and F0 predictors used for content replacement | Sec. 2.3 |
| `AudioLM/audiolm_english.pt`, `config.yaml` | Autoregressive decoder from articulatory features to RVQ indices (Stream B) | Sec. 2.1.2 |
| `BigVGAN/generator_english`, `config.json` | BigVGAN generator conditioned on RVQ reconstruction, F0 and x-vector (1473 dims, 16 kHz) | Sec. 2.4 |
| `POOL/english/` | Pseudospeaker pool: ECAPA-TDNN x-vectors, gender map, Affinity Propagation cluster indices, pitch statistics | Sec. 2.4 |
| `*_french*`, `rvq_model_fr.pth`, `POOL/french/` | French counterparts of the models and pool above | Not evaluated in the paper |
| `SHA256SUMS` | Checksums for every file | |
The French checkpoints are provided as is. The paper evaluates English only,
and the command-line tools in the code repository load the English models by
default.
## Training data
- **RVQ-Whisper:** Common Voice 17, LibriTTS, TED-LIUM and VoxPopuli (English), filtered with wv-mos > 3.3 and < 10 min per speaker.
- **Token decoder and BigVGAN:** Common Voice 17, LibriTTS *train-clean-100*, VoxPopuli, VCTK and LibriSpeech *train-clean-360* for English. The French models use Common Voice 17, Multilingual LibriSpeech, VoxPopuli and VoxForge.
- **Speaker pools:** ECAPA-TDNN x-vectors of Common Voice speakers held out from all evaluation sets (English: 1,465 speakers; French: 251 speakers). Speaker IDs are Common Voice's hashed client IDs. No audio or transcripts are included.
## License
The weights and the speaker pool are released under
[CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/): you may use
them for research and other non-commercial purposes, with attribution. Some of
the training corpora carry non-commercial terms. The code is MIT-licensed; see
the GitHub repository. For commercial use, contact [Nijta](https://nijta.com).
## Intended use and limitations
These models are for **protecting** speakers when recordings are shared or
reused, and for research on speech privacy.
- Do not use them to impersonate people, or to make synthetic speech look like a real person's recording.
- Do not try to re-identify the speakers behind the pool x-vectors.
- The content protection only covers what the NER tagger finds. Review anonymized data before release.
- Emotion preservation is limited (UAR 40.11% on IEMOCAP).
## Citation
```bibtex
@inproceedings{golmakani2026acoustic,
title = {Acoustic token admixture for joint speaker and content anonymization},
author = {Golmakani, Ali and Hosseini, Seyed Ahmad and Bendali, Omar Manil and Vincent, Emmanuel and Srivastava, Brij Mohan Lal},
booktitle = {Proc. Interspeech 2026},
year = {2026}
}
```