AURA-1.3B: AUdio-dRiven streaming Avatar (1.3B)
The 1.3B model of the AURA family, and the only one that streams in real time (576p; in the paper, 20.7 fps end-to-end on a single H100 for 16 Hz playback, 23Γ faster than its teacher). It turns a portrait image and a speech audio stream into a talking-head video, chunk by chunk, with a rolling KV cache. The 4-step causal student is distilled from AURA-1.3B-Teacher-TalkVerse, a bidirectional 1.3B teacher conditioned on Mimi audio tokens.
Project page Β· Paper Β· Code Β· AURA-5B
Nathaniel Cohen1,2, Nicolas Dufour1, AmΓ©lie Royer1, Alasdair Newson2, Patrick PΓ©rez3* 1Kyutai Β· 2Sorbonne UniversitΓ©, ISIR Β· 3Google DeepMind Β· *work done while at Kyutai
Samples
Generated by AURA-1.3B from the same speech with three different reference images and prompts. More samples are on the project page.
Model details
| Base model | Wan2.1-T2V-1.3B |
| Teacher | kyutai/AURA-1.3B-Teacher-TalkVerse |
| Audio encoder | Mimi |
| Format | Each checkpoint comes in two forms. *_merged: standalone fp32 safetensors (~6.3 GB), Wan2.1-T2V-1.3B with the teacher and student LoRAs (rank 128, alpha 128) merged in, plus the Mimi audio modules; only the VAE and T5 text encoder are still needed from Wan2.1-T2V-1.3B. *_lora_only: the raw training checkpoint (LoRA + trained modules), to apply on top of Wan2.1-T2V-1.3B, the teacher and the earlier stages. |
| Attention | Block-wise causal over chunks of 4 latent frames (β0.64 s) |
| Inference | 4 denoising steps / chunk, rolling KV cache Β· 576p Β· 20.7 fps end-to-end on one H100 in the paper (DiT 28.8 fps, VAE 96.9 fps) |
Files
LoRA-only and merged checkpoints. Every training stage fine-tunes LoRA adapters (rank 128, alpha 128) plus a few non-LoRA modules (Mimi audio modules). We release each checkpoint in both forms: *_lora_only is the raw training checkpoint (LoRA updates and trained modules only, to apply on top of Wan2.1-T2V-1.3B, the teacher and the earlier stages), and *_merged is a standalone model with all LoRA updates merged in.
| File | Description |
|---|---|
stage2_dmd/step-6500_ema_merged.safetensors |
Final AURA-1.3B model: stage 2, DMD, EMA weights, standalone. Use this one. |
stage2_dmd/step-6500_ema_lora_only.safetensors |
Same, LoRA-only form. Apply on Wan2.1-T2V-1.3B β teacher (step-60000_lora_only in AURA-1.3B-Teacher-TalkVerse) β stage1_ode/step-40000_lora_only β this |
stage1_ode/step-40000_merged.safetensors |
Stage-1 ODE-distilled student (before DMD), standalone |
stage1_ode/step-40000_lora_only.safetensors |
Same, LoRA-only form (apply on Wan2.1-T2V-1.3B β teacher β this) |
{stage2_dmd,stage1_ode}/launch_config.json |
Training arguments, launch command, seed and code commit |
{stage2_dmd,stage1_ode}/step-*.metrics.jsonl |
Training metrics (loss, val/critic losses, LRβ¦) for that checkpoint, one JSON record per logged step up to the checkpoint step; resumed launches stitched together |
Training
AURA-1.3B uses the same recipe as AURA-5B:
- Stage 1: ODE distillation. The student is trained with an MSE loss on 30K teacher ODE trajectories generated by AURA-1.3B-Teacher-TalkVerse (Mimi audio, 80 frames). These trajectories are released in the AURA dataset.
- Stage 2: DMD. Distribution Matching Distillation with backward simulation, run with a frozen audio LoRA. It uses a learning rate of 2e-7 and EMA, and the EMA weights are the ones released.
Usage
git clone https://github.com/kyutai-labs/aura && cd aura
uv sync
uv run generate.py --image portrait.jpg --audio speech.wav --output out.mp4 # --model aura-1.3b is the default
The script downloads everything it needs from the Hub: stage2_dmd/step-6500_ema_merged.safetensors from this repo (standalone DiT with the Mimi audio modules), the VAE and T5 text encoder from Wan2.1-T2V-1.3B, and Mimi for audio features.
It samples 4 steps per chunk (timesteps 1000, 745, 498, 255) without classifier-free guidance, at 16 fps, in 576Γ576 or 768Γ512. Other options: --prompt, --num_chunks, --seed, --compile.
With --compile, this reference code runs at about 16 fps on one H100, i.e. real time for 16 fps playback; the 20.7 fps reported in the paper come from an optimized pipeline.
The other checkpoints (stage1_ode/*, *_lora_only) are released for research and are not loaded by the reference code.
Evaluation
| Benchmark | FID β | FVD β | JEDi β | CSIM β | Sync-C β | Sync-D β | WE β |
|---|---|---|---|---|---|---|---|
| HALLO3 test (100 clips, 10 s) | 56.8 | 177.1 | 0.403 | 0.84 | 4.07 | 9.93 | 0.0036 |
| CelebV-HQ test (150 clips) | 51.5 | 248.8 | 0.574 | 0.77 | 4.76 | 10.1 | 0.0046 |
AURA-1.3B has the best identity preservation (CSIM) of all compared methods on both test sets. See the paper for the full tables.
License
AURA-1.3B starts from Wan2.1-T2V-1.3B: its teacher, AURA-1.3B-Teacher-TalkVerse, is Wan2.1-T2V-1.3B fine-tuned with LoRA and Mimi audio modules, and AURA-1.3B is distilled from that teacher. The weights (merged and LoRA-only) are released under CC BY-NC 4.0: non-commercial use only. This follows the terms of the training data: the student is distilled on teacher trajectories conditioned on reference images and prompts from the Hallo3 training data (CC BY-NC-ND 4.0), see the AURA dataset. The VAE and T5 text encoder you download from Wan2.1-T2V-1.3B keep their own license (Apache 2.0).
Responsible use
The model generates realistic videos of real-looking people speaking arbitrary audio. Do not use it to impersonate people, create non-consensual content or produce disinformation. Only animate portraits for which you have the subject's consent, and disclose that videos are AI-generated.
Contact
For questions, contact Nathaniel Cohen (nathaniel.cohen@kyutai.org) or AmΓ©lie Royer (amelie@kyutai.org).
Citation
@inproceedings{cohen2026aura,
title = {{AURA}: {AU}dio-d{R}iven streaming {A}vatar},
author = {Cohen, Nathaniel and Dufour, Nicolas and Royer, Am{\'e}lie
and Newson, Alasdair and P{\'e}rez, Patrick},
booktitle = {British Machine Vision Conference (BMVC)},
year = {2026},
}
Model tree for kyutai/AURA-1.3B
Base model
Wan-AI/Wan2.1-T2V-1.3B