mlr2000/vocoder-small

VocBulwark HiFi-GAN — watermarking neural vocoder (inference export)

Speaker-conditioned BigVGAN / HiFi-GAN neural vocoder. It turns an input mel-spectrogram (what is said) into a 24 kHz waveform, conditioned on a precomputed 768-d speaker embedding (whose voice). Every clip it generates carries a fixed 32-bit provenance watermark for provenance — see Watermark.

This is the lean, inference-only vocoder: the frozen perceptual-loss base models (Whisper / WavLM / Wav2Vec2), the training discriminators, and the speaker encoder have all been stripped — you pass the speaker embedding in. Use the companion speaker-encoder repo to turn a reference clip into that embedding. The modeling code is bundled, so it loads with trust_remote_code=True without the training repo.

For higher watermark capacity (50-bit) and a larger generator (1536 initial channels vs 512), see the large model variant mlr2000/vocoder-large.

Companion Models

This model is part of a set of 6 repositories:

Repo	Role
mlr2000/vocoder-large	Large vocoder
mlr2000/vocoder-large-watermark-detector	Watermark detector for the large model
mlr2000/vocoder-large-speaker-encoder	Speaker encoder for the large model
mlr2000/vocoder-small	Small vocoder (this repo)
mlr2000/vocoder-small-watermark-detector	Watermark detector for the small model
mlr2000/vocoder-small-speaker-encoder	Speaker encoder for the small model

Model summary

Architecture	HiFiGANArchitecture (BigVGAN generator, snakebeta activation)
Inputs	log-mel spectrogram (96 mel channels) + speaker embedding (768-d)
Output	mono waveform, 24 kHz
Speaker conditioning	precomputed embedding (from the companion speaker encoder)
Generator	initial channels 512, upsample rates [4, 4, 2, 2, 2, 2]
Watermark	32-bit fixed VocBulwark signature, always embedded
Framework	🤗 Transformers, PyTorch, safetensors

Usage

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained("mlr2000/vocoder-small", trust_remote_code=True).eval()

mel = torch.randn(1, model.config.hifigan_in_channels, 200)      # [B, mel, T]
emb = torch.randn(1, model.config.speaker_embedding_size)        # [B, 768] from the speaker encoder
with torch.no_grad():
    audio = model(mel_spectrogram=mel, speaker_embedding=emb).audio
# audio: [B, 1, samples] @ model.config.target_sample_rate

See example_roundtrip.ipynb in this repo for the full pipeline (reference clip → speaker encoder → embedding → vocode → verify watermark).

Training

Training data	Multilingual LibriSpeech (8 languages, ~22,200h) and Common Voice (14 languages, ~3,000h)
Training steps	1,000,000
Hardware	2 × NVIDIA H100 PCIe GPUs
Training objective	Discriminator-free: mel spectrogram + WavLM + wav2vec 2.0 + Whisper encoder losses
Effective batch size	32
Learning rate	1e-3

Watermark

Every clip this model generates carries a fixed 32-bit provenance watermark (config.fixed_watermark). It is embedded automatically inside forward and cannot be disabled or changed through this interface — there is deliberately no watermark argument to override. To check whether a given audio came from this model, use the companion detector repo, which extracts the bits and compares them to the same fixed code.

To verify whether a given audio clip was generated by this model, use the companion detector repo (mlr2000/vocoder-small-watermark-detector), which extracts the embedded bits and compares them to the known fixed code.

Notes

• Inputs: log-mel spectrogram (config.hifigan_in_channels channels) and a [B, config.speaker_embedding_size] speaker embedding.
• Output: mono waveform at config.target_sample_rate.
• Use the speaker embedding from the speaker encoder this vocoder was trained with. A mismatched encoder will not condition it correctly.
• Not intended for voice cloning of real individuals without consent, or any deceptive or impersonation use.

Citation

If you use this model, please cite:

@misc{muletta2026,
  title  = {Training a Discriminator-Free Foundation Vocoder 
             with Integrated Audio Watermarking},
  author = {Muletta, Romolo and Deriu, Jan},
  year   = {2026},
  note   = {VT2 Project Report, ZHAW School of Engineering}
}

License

cc-by-4.0. Trained on MLS (CC-BY-4.0) and Common Voice (CC0); builds on BigVGAN (MIT) and wav2vec 2.0 (Apache-2.0). Please retain attribution when redistributing or building on this model.