DAC-VAE 1024d @ 6.25 Hz
A frozen DAC-VAE audio encoder (latent dim 1024, frame rate 6.25 Hz) attached to a Qwen3.5AE-4B language model. Part of a 4x4 ablation over latent dimension and frame rate.
Only the 4-layer audio projector (~18M params, 0.43% of the model) is trained; the DAC-VAE encoder and the language model are frozen.
Input format
Audio replaces <|audio_pad|> spans; a correspond token selects the task.
ASR : "Transcribe the speech." [audio] <|asr_correspond|> -> transcript <|im_end|>
QA : "Answer the speech." [audio] <|answer|> -> answer <|im_end|>
Results
| Benchmark | Score |
|---|---|
| LibriSpeech test-clean WER | 4.68% |
| TriviaQA Speech Cover-EM | 26.58% |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch, soundfile as sf, torchaudio
M = "REPO_ID"
tok = AutoTokenizer.from_pretrained(M, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
M, trust_remote_code=True, dtype=torch.bfloat16,
attn_implementation="flash_attention_2").cuda().eval()
wav, sr = sf.read("speech.wav", dtype="float32", always_2d=True)
w = torch.from_numpy(wav.T)
if w.shape[0] > 1: w = w.mean(0, keepdim=True)
sr_target = model.config.audio_config.dac_sample_rate # 48000
if sr != sr_target: w = torchaudio.functional.resample(w, sr, sr_target)
hop = round(sr_target / model.config.audio_config.dac_latent_hz)
T = w.shape[-1] // hop
w = w[:, :T * hop]
AP = tok.convert_tokens_to_ids("<|audio_pad|>")
CORR = tok.convert_tokens_to_ids("<|asr_correspond|>") # or <|answer|>
ids = tok.encode("Transcribe the speech.", add_special_tokens=False) + [AP] * T + [CORR]
out = model.generate(
input_ids=torch.tensor([ids]).cuda(),
audio_features=w.unsqueeze(0).cuda().to(torch.bfloat16),
audio_lengths=torch.tensor([T]).cuda(),
max_new_tokens=200, do_sample=False)
print(tok.decode(out[0, len(ids):], skip_special_tokens=True))
Audio is encoded per utterance (audio_lengths), matching training.
Training
40,000 steps, effective batch 64 sequences. ASR (15.5M utterances) with speech-QA
(InstructS2S-200K) mixed in from step 20,000. Sequence packing with cross-sample
isolation; cutoff_len is set per frame rate so every variant sees the same number
of utterances per step.
- Downloads last month
- -