Instructions to use aaatmy/pe-av-small-audio-ONNX with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PerceptionEncoder
How to use aaatmy/pe-av-small-audio-ONNX with PerceptionEncoder:
# Use any PE model as a vision encoder import core.vision_encoder.pe as pe model = pe.VisionTransformer.from_config("aaatmy/pe-av-small-audio-ONNX", pretrained=True) - Notebooks
- Google Colab
- Kaggle
PE-AV small: audio encoder only (ONNX)
ONNX export of the audio branch of Meta's facebook/pe-av-small (Perception Encoder Audio-Visual). This is an unofficial conversion; it is not affiliated with or endorsed by Meta.
Included: the audio encoder (DAC encoder, CLS token, transformer) plus audio_head, which
maps 48 kHz audio to an embedding in PE-AV's audio–text space. About 117 M parameters.
Not included: the text encoder and the video encoder. To score audio against text,
compute text embeddings once with the original model (see "Text side" below) and store them.
| File | Weights | Size |
|---|---|---|
onnx/model.onnx |
float32 | 470 MB |
onnx/model_q8.onnx |
8-bit block-wise (block 32), weight-only MatMul; the convolutional DAC encoder stays float32 |
222 MB |
Input and output
| Name | Shape | Type |
|---|---|---|
input_values |
[batch, 1, samples] |
float32, 48 kHz mono |
audio_embedding (output) |
[batch, 1024] |
float32, not normalised |
Preprocessing matches PeAudioFeatureExtractor (preprocessor_config.json) for a single clip:
resample to 48 kHz mono, then reflect-pad the end to a multiple of 1920 samples. Feed
clips one at a time, or batch clips of equal length. Padding masks are not supported.
Scoring (PE-AV uses sigmoid, SigLIP-style logits; values in scoring.json):
logit = audio_embedding · text_embedding * 2.205655813217163 + (-10.120551109313965)
Because the scale is positive and the bias is shared, ranking classes by the raw dot product gives the same order.
Usage
import numpy as np, onnxruntime as ort, soundfile as sf, soxr
from huggingface_hub import hf_hub_download
path = hf_hub_download("aaatmy/pe-av-small-audio-ONNX", "onnx/model_q8.onnx")
sess = ort.InferenceSession(path, providers=["CPUExecutionProvider"])
def embed(wav_path, seconds=10):
x, sr = sf.read(wav_path, dtype="float32", always_2d=True)
x = soxr.resample(x.mean(axis=1), sr, 48000)[: 48000 * seconds]
pad = (-len(x)) % 1920
if pad:
x = np.pad(x, (0, pad), mode="reflect")
return sess.run(["audio_embedding"], {"input_values": x[None, None, :].astype(np.float32)})[0][0]
emb = embed("track.wav") # shape (1024,)
# scores = text_prototypes @ emb # text_prototypes: [n_classes, 1024] from the original text encoder
Text side (computed offline with the original model)
With transformers 5.20, PeAudioVideoModel.get_text_audio_embeds() fails: it calls a
method that PeAudioModel does not have. Use the same path as the model's forward():
from transformers import AutoTokenizer, PeAudioVideoModel
model = PeAudioVideoModel.from_pretrained("facebook/pe-av-small", ignore_mismatched_sizes=True).eval()
tok = AutoTokenizer.from_pretrained("facebook/pe-av-small")
t = tok(["the sound of a violin", "the sound of drums"], padding=True, return_tensors="pt")
h = model.text_model(**t, output_hidden_states=True).hidden_states[-1][:, 0]
text_embeddings = model.audio_model.text_audio_head(h)
ignore_mismatched_sizes=True is needed because, in transformers 5.20, one video weight
(video_model.video_encoder.embedder.proj.weight) is rebuilt with a different shape. No
audio or text weight is affected. The text encoder tokenizer is English-only, so write prompts in English.
Verification
Parity with PyTorch. We compared each file with PeAudioVideoModel.get_audio_embeds()
(transformers 5.20.2), using the same feature-extractor input, on real recordings of several
lengths. "Top-1 agree" means both models pick the same class among 8 instrument prototypes.
| File | Clips | min cosine | mean cosine | Top-1 agree |
|---|---|---|---|---|
model.onnx |
100 (3 s, 4 s, 10 s, 30 s) | 1.000000 | 1.000000 | 100 % |
model_q8.onnx |
250 (3 s, 4 s, 10 s, 30 s) | 0.99944 | 0.99978 – 0.99990 | 100 % on 3 s, 10 s and 30 s solos; 97 % on 4 s NSynth notes |
On NSynth notes the 8 Medley-solos classes are out of domain, so their scores are close and small changes can flip the top class.
Rejected variant: accuracy_level=4 (int8 compute in MatMulNBits) reached a minimum
cosine of only 0.9966. It is not included here.
Zero-shot instrument recognition with the original PyTorch model. These are our measurements; ONNX parity above means the ONNX files give the same results. Text prompts were averaged per class.
| Dataset | Classes | Accuracy | Balanced accuracy |
|---|---|---|---|
| NSynth test (4,096 notes) | 10 instrument families | 75.1 % | 77.6 % |
| Medley-solos-DB test (12,236 clips) | 8 instruments | 95.4 % | 93.1 % |
CPU latency (Apple Silicon, onnxruntime 1.31): about 2.8 s per 10 s segment and 0.85 s per 3 s segment, for both files. Most of the time is spent in the convolutional DAC encoder, which is not quantised. The CoreML execution provider needs a fixed input shape. With one, CPU plus Neural Engine took 2.3 s per 10 s segment, while GPU configurations were slower than CPU.
How it was produced
Scripts are in export/:
export_peav_audio.py: TorchScript ONNX export (opset 17), transformers 5.20.2, eager attention. It exports without a padding mask, which also skips transformers'vmap-based mask builder, and writesscoring.json.quantize_int8.py block8: onnxruntimeMatMulNBitsQuantizer, 8 bits, block size 32, symmetric,MatMulonly.
SHA256SUMS lists checksums for every file.
License and attribution
Apache License 2.0. LICENSE is copied from Meta's
perception_models repository (LICENSE.PE).
Original model: Meta FAIR, facebook/pe-av-small.
Modifications: extraction of the audio branch (encoder + audio_head), ONNX export, and an
8-bit weight-only quantised variant.
@misc{vyas2025pushingfrontieraudiovisualperception,
title={Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning},
author={Apoorv Vyas and Heng-Jui Chang and Cheng-Fu Yang and Po-Yao Huang and Luya Gao and Julius Richter and Sanyuan Chen and Matt Le and Piotr Dollár and Christoph Feichtenhofer and Ann Lee and Wei-Ning Hsu},
year={2025},
eprint={2512.19687},
archivePrefix={arXiv},
primaryClass={cs.SD},
url={https://arxiv.org/abs/2512.19687},
}
Model tree for aaatmy/pe-av-small-audio-ONNX
Base model
facebook/pe-av-small