|
Download README.md from nijta/acoustic-token-admixture: direct link, hf CLI and curl.
- Browser
- Download file 4.59 kB
-
https://huggingface.co/nijta/acoustic-token-admixture/resolve/main/README.md
- Command line
-
hf download hf://nijta/acoustic-token-admixture/README.md
-
curl -L -H "Authorization: Bearer $HF_TOKEN" -o README.md https://huggingface.co/nijta/acoustic-token-admixture/resolve/main/README.md
4.59 kB
| license: cc-by-nc-4.0 | |
| language: | |
| - en | |
| - fr | |
| library_name: pytorch | |
| pipeline_tag: audio-to-audio | |
| tags: | |
| - speaker-anonymization | |
| - voice-privacy | |
| - speech-editing | |
| - residual-vector-quantization | |
| - bigvgan | |
| - interspeech2026 | |
| # Acoustic Token Admixture: model weights and speaker pool | |
| Pretrained checkpoints and the pseudospeaker pool for the Interspeech 2026 paper | |
| **"Acoustic token admixture for joint speaker and content anonymization"** | |
| (Ali Golmakani, Seyed Ahmad Hosseini, Omar Manil Bendali, Emmanuel Vincent, Brij Mohan Lal Srivastava). | |
| - **Code:** <https://github.com/Nijta/acoustic-token-admixture> | |
| - **Audio samples:** <https://nijta.github.io/acoustic-token-admixture/> | |
| The system anonymizes both the speaker's voice and named entities in the | |
| content. It mixes encoder-derived and phoneme-conditioned RVQ tokens frame by | |
| frame, then resynthesizes the result with a BigVGAN vocoder conditioned on a | |
| pseudospeaker. On VoicePrivacy 2024 it reaches EER 42.54% with WER 3.73% | |
| (β = 0.7). | |
| ## Quick start | |
| ```bash | |
| git clone https://github.com/Nijta/acoustic-token-admixture.git | |
| cd acoustic-token-admixture | |
| pip install -r requirements.txt | |
| hf download nijta/acoustic-token-admixture --local-dir models | |
| (cd models && sha256sum -c SHA256SUMS) | |
| export MODELS_DIR=$PWD/models | |
| export PYTHONPATH=".:./audiolm:./bigvgan/src" | |
| python static_infer.py --inputs input.wav --output-dir outputs --seeds 52 \ | |
| --gender any --admixture-ratio 0.7 --pitch-f 0.75 --db 2 | |
| ``` | |
| The folder layout matches what the code expects in `$MODELS_DIR`. | |
| ## Files | |
| | Path | Component | Paper | | |
| |---|---|---| | |
| | `RVQWhisper/rvq_model_en.pth`, `config.yaml` | 8 x 1024 residual vector quantizer on frozen Whisper large-v2 encoder features (Stream A) | Sec. 2.1.1 | | |
| | `Aligner/aligner_english.pth` | CTC phoneme head (111 classes) for forced alignment | Sec. 2.1.2 | | |
| | `Aligner/dur_predictor.pt`, `f0_predictor.pth` | Duration and F0 predictors used for content replacement | Sec. 2.3 | | |
| | `AudioLM/audiolm_english.pt`, `config.yaml` | Autoregressive decoder from articulatory features to RVQ indices (Stream B) | Sec. 2.1.2 | | |
| | `BigVGAN/generator_english`, `config.json` | BigVGAN generator conditioned on RVQ reconstruction, F0 and x-vector (1473 dims, 16 kHz) | Sec. 2.4 | | |
| | `POOL/english/` | Pseudospeaker pool: ECAPA-TDNN x-vectors, gender map, Affinity Propagation cluster indices, pitch statistics | Sec. 2.4 | | |
| | `*_french*`, `rvq_model_fr.pth`, `POOL/french/` | French counterparts of the models and pool above | Not evaluated in the paper | | |
| | `SHA256SUMS` | Checksums for every file | | | |
| The French checkpoints are provided as is. The paper evaluates English only, | |
| and the command-line tools in the code repository load the English models by | |
| default. | |
| ## Training data | |
| - **RVQ-Whisper:** Common Voice 17, LibriTTS, TED-LIUM and VoxPopuli (English), filtered with wv-mos > 3.3 and < 10 min per speaker. | |
| - **Token decoder and BigVGAN:** Common Voice 17, LibriTTS *train-clean-100*, VoxPopuli, VCTK and LibriSpeech *train-clean-360* for English. The French models use Common Voice 17, Multilingual LibriSpeech, VoxPopuli and VoxForge. | |
| - **Speaker pools:** ECAPA-TDNN x-vectors of Common Voice speakers held out from all evaluation sets (English: 1,465 speakers; French: 251 speakers). Speaker IDs are Common Voice's hashed client IDs. No audio or transcripts are included. | |
| ## License | |
| The weights and the speaker pool are released under | |
| [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/): you may use | |
| them for research and other non-commercial purposes, with attribution. Some of | |
| the training corpora carry non-commercial terms. The code is MIT-licensed; see | |
| the GitHub repository. For commercial use, contact [Nijta](https://nijta.com). | |
| ## Intended use and limitations | |
| These models are for **protecting** speakers when recordings are shared or | |
| reused, and for research on speech privacy. | |
| - Do not use them to impersonate people, or to make synthetic speech look like a real person's recording. | |
| - Do not try to re-identify the speakers behind the pool x-vectors. | |
| - The content protection only covers what the NER tagger finds. Review anonymized data before release. | |
| - Emotion preservation is limited (UAR 40.11% on IEMOCAP). | |
| ## Citation | |
| ```bibtex | |
| @inproceedings{golmakani2026acoustic, | |
| title = {Acoustic token admixture for joint speaker and content anonymization}, | |
| author = {Golmakani, Ali and Hosseini, Seyed Ahmad and Bendali, Omar Manil and Vincent, Emmanuel and Srivastava, Brij Mohan Lal}, | |
| booktitle = {Proc. Interspeech 2026}, | |
| year = {2026} | |
| } | |
| ``` | |