PE-AV small: audio encoder only (ONNX)

ONNX export of the audio branch of Meta's facebook/pe-av-small (Perception Encoder Audio-Visual). This is an unofficial conversion; it is not affiliated with or endorsed by Meta.

Included: the audio encoder (DAC encoder, CLS token, transformer) plus audio_head, which maps 48 kHz audio to an embedding in PE-AV's audio–text space. About 117 M parameters. Not included: the text encoder and the video encoder. To score audio against text, compute text embeddings once with the original model (see "Text side" below) and store them.

File Weights Size
onnx/model.onnx float32 470 MB
onnx/model_q8.onnx 8-bit block-wise (block 32), weight-only MatMul; the convolutional DAC encoder stays float32 222 MB

Input and output

Name Shape Type
input_values [batch, 1, samples] float32, 48 kHz mono
audio_embedding (output) [batch, 1024] float32, not normalised

Preprocessing matches PeAudioFeatureExtractor (preprocessor_config.json) for a single clip: resample to 48 kHz mono, then reflect-pad the end to a multiple of 1920 samples. Feed clips one at a time, or batch clips of equal length. Padding masks are not supported.

Scoring (PE-AV uses sigmoid, SigLIP-style logits; values in scoring.json):

logit = audio_embedding · text_embedding * 2.205655813217163 + (-10.120551109313965)

Because the scale is positive and the bias is shared, ranking classes by the raw dot product gives the same order.

Usage

import numpy as np, onnxruntime as ort, soundfile as sf, soxr
from huggingface_hub import hf_hub_download

path = hf_hub_download("aaatmy/pe-av-small-audio-ONNX", "onnx/model_q8.onnx")
sess = ort.InferenceSession(path, providers=["CPUExecutionProvider"])

def embed(wav_path, seconds=10):
    x, sr = sf.read(wav_path, dtype="float32", always_2d=True)
    x = soxr.resample(x.mean(axis=1), sr, 48000)[: 48000 * seconds]
    pad = (-len(x)) % 1920
    if pad:
        x = np.pad(x, (0, pad), mode="reflect")
    return sess.run(["audio_embedding"], {"input_values": x[None, None, :].astype(np.float32)})[0][0]

emb = embed("track.wav")            # shape (1024,)
# scores = text_prototypes @ emb    # text_prototypes: [n_classes, 1024] from the original text encoder

Text side (computed offline with the original model)

With transformers 5.20, PeAudioVideoModel.get_text_audio_embeds() fails: it calls a method that PeAudioModel does not have. Use the same path as the model's forward():

from transformers import AutoTokenizer, PeAudioVideoModel
model = PeAudioVideoModel.from_pretrained("facebook/pe-av-small", ignore_mismatched_sizes=True).eval()
tok = AutoTokenizer.from_pretrained("facebook/pe-av-small")
t = tok(["the sound of a violin", "the sound of drums"], padding=True, return_tensors="pt")
h = model.text_model(**t, output_hidden_states=True).hidden_states[-1][:, 0]
text_embeddings = model.audio_model.text_audio_head(h)

ignore_mismatched_sizes=True is needed because, in transformers 5.20, one video weight (video_model.video_encoder.embedder.proj.weight) is rebuilt with a different shape. No audio or text weight is affected. The text encoder tokenizer is English-only, so write prompts in English.

Verification

Parity with PyTorch. We compared each file with PeAudioVideoModel.get_audio_embeds() (transformers 5.20.2), using the same feature-extractor input, on real recordings of several lengths. "Top-1 agree" means both models pick the same class among 8 instrument prototypes.

File Clips min cosine mean cosine Top-1 agree
model.onnx 100 (3 s, 4 s, 10 s, 30 s) 1.000000 1.000000 100 %
model_q8.onnx 250 (3 s, 4 s, 10 s, 30 s) 0.99944 0.99978 – 0.99990 100 % on 3 s, 10 s and 30 s solos; 97 % on 4 s NSynth notes

On NSynth notes the 8 Medley-solos classes are out of domain, so their scores are close and small changes can flip the top class.

Rejected variant: accuracy_level=4 (int8 compute in MatMulNBits) reached a minimum cosine of only 0.9966. It is not included here.

Zero-shot instrument recognition with the original PyTorch model. These are our measurements; ONNX parity above means the ONNX files give the same results. Text prompts were averaged per class.

Dataset Classes Accuracy Balanced accuracy
NSynth test (4,096 notes) 10 instrument families 75.1 % 77.6 %
Medley-solos-DB test (12,236 clips) 8 instruments 95.4 % 93.1 %

CPU latency (Apple Silicon, onnxruntime 1.31): about 2.8 s per 10 s segment and 0.85 s per 3 s segment, for both files. Most of the time is spent in the convolutional DAC encoder, which is not quantised. The CoreML execution provider needs a fixed input shape. With one, CPU plus Neural Engine took 2.3 s per 10 s segment, while GPU configurations were slower than CPU.

How it was produced

Scripts are in export/:

  1. export_peav_audio.py: TorchScript ONNX export (opset 17), transformers 5.20.2, eager attention. It exports without a padding mask, which also skips transformers' vmap-based mask builder, and writes scoring.json.
  2. quantize_int8.py block8: onnxruntime MatMulNBitsQuantizer, 8 bits, block size 32, symmetric, MatMul only.

SHA256SUMS lists checksums for every file.

License and attribution

Apache License 2.0. LICENSE is copied from Meta's perception_models repository (LICENSE.PE). Original model: Meta FAIR, facebook/pe-av-small. Modifications: extraction of the audio branch (encoder + audio_head), ONNX export, and an 8-bit weight-only quantised variant.

@misc{vyas2025pushingfrontieraudiovisualperception,
      title={Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning},
      author={Apoorv Vyas and Heng-Jui Chang and Cheng-Fu Yang and Po-Yao Huang and Luya Gao and Julius Richter and Sanyuan Chen and Matt Le and Piotr Dollár and Christoph Feichtenhofer and Ann Lee and Wei-Ning Hsu},
      year={2025},
      eprint={2512.19687},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2512.19687},
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for aaatmy/pe-av-small-audio-ONNX

Quantized
(1)
this model

Space using aaatmy/pe-av-small-audio-ONNX 1

Paper for aaatmy/pe-av-small-audio-ONNX