AURA-1.3B: AUdio-dRiven streaming Avatar (1.3B)

The 1.3B model of the AURA family, and the only one that streams in real time (576p; in the paper, 20.7 fps end-to-end on a single H100 for 16 Hz playback, 23Γ— faster than its teacher). It turns a portrait image and a speech audio stream into a talking-head video, chunk by chunk, with a rolling KV cache. The 4-step causal student is distilled from AURA-1.3B-Teacher-TalkVerse, a bidirectional 1.3B teacher conditioned on Mimi audio tokens.

Project page Β· Paper Β· Code Β· AURA-5B

Nathaniel Cohen1,2, Nicolas Dufour1, AmΓ©lie Royer1, Alasdair Newson2, Patrick PΓ©rez3* 1Kyutai Β· 2Sorbonne UniversitΓ©, ISIR Β· 3Google DeepMind Β· *work done while at Kyutai

Samples

Generated by AURA-1.3B from the same speech with three different reference images and prompts. More samples are on the project page.

Model details

Base model Wan2.1-T2V-1.3B
Teacher kyutai/AURA-1.3B-Teacher-TalkVerse
Audio encoder Mimi
Format Each checkpoint comes in two forms. *_merged: standalone fp32 safetensors (~6.3 GB), Wan2.1-T2V-1.3B with the teacher and student LoRAs (rank 128, alpha 128) merged in, plus the Mimi audio modules; only the VAE and T5 text encoder are still needed from Wan2.1-T2V-1.3B. *_lora_only: the raw training checkpoint (LoRA + trained modules), to apply on top of Wan2.1-T2V-1.3B, the teacher and the earlier stages.
Attention Block-wise causal over chunks of 4 latent frames (β‰ˆ0.64 s)
Inference 4 denoising steps / chunk, rolling KV cache Β· 576p Β· 20.7 fps end-to-end on one H100 in the paper (DiT 28.8 fps, VAE 96.9 fps)

Files

LoRA-only and merged checkpoints. Every training stage fine-tunes LoRA adapters (rank 128, alpha 128) plus a few non-LoRA modules (Mimi audio modules). We release each checkpoint in both forms: *_lora_only is the raw training checkpoint (LoRA updates and trained modules only, to apply on top of Wan2.1-T2V-1.3B, the teacher and the earlier stages), and *_merged is a standalone model with all LoRA updates merged in.

File Description
stage2_dmd/step-6500_ema_merged.safetensors Final AURA-1.3B model: stage 2, DMD, EMA weights, standalone. Use this one.
stage2_dmd/step-6500_ema_lora_only.safetensors Same, LoRA-only form. Apply on Wan2.1-T2V-1.3B β†’ teacher (step-60000_lora_only in AURA-1.3B-Teacher-TalkVerse) β†’ stage1_ode/step-40000_lora_only β†’ this
stage1_ode/step-40000_merged.safetensors Stage-1 ODE-distilled student (before DMD), standalone
stage1_ode/step-40000_lora_only.safetensors Same, LoRA-only form (apply on Wan2.1-T2V-1.3B β†’ teacher β†’ this)
{stage2_dmd,stage1_ode}/launch_config.json Training arguments, launch command, seed and code commit
{stage2_dmd,stage1_ode}/step-*.metrics.jsonl Training metrics (loss, val/critic losses, LR…) for that checkpoint, one JSON record per logged step up to the checkpoint step; resumed launches stitched together

Training

AURA-1.3B uses the same recipe as AURA-5B:

  1. Stage 1: ODE distillation. The student is trained with an MSE loss on 30K teacher ODE trajectories generated by AURA-1.3B-Teacher-TalkVerse (Mimi audio, 80 frames). These trajectories are released in the AURA dataset.
  2. Stage 2: DMD. Distribution Matching Distillation with backward simulation, run with a frozen audio LoRA. It uses a learning rate of 2e-7 and EMA, and the EMA weights are the ones released.

Usage

git clone https://github.com/kyutai-labs/aura && cd aura
uv sync
uv run generate.py --image portrait.jpg --audio speech.wav --output out.mp4   # --model aura-1.3b is the default

The script downloads everything it needs from the Hub: stage2_dmd/step-6500_ema_merged.safetensors from this repo (standalone DiT with the Mimi audio modules), the VAE and T5 text encoder from Wan2.1-T2V-1.3B, and Mimi for audio features. It samples 4 steps per chunk (timesteps 1000, 745, 498, 255) without classifier-free guidance, at 16 fps, in 576Γ—576 or 768Γ—512. Other options: --prompt, --num_chunks, --seed, --compile. With --compile, this reference code runs at about 16 fps on one H100, i.e. real time for 16 fps playback; the 20.7 fps reported in the paper come from an optimized pipeline.

The other checkpoints (stage1_ode/*, *_lora_only) are released for research and are not loaded by the reference code.

Evaluation

Benchmark FID ↓ FVD ↓ JEDi ↓ CSIM ↑ Sync-C ↑ Sync-D ↓ WE ↓
HALLO3 test (100 clips, 10 s) 56.8 177.1 0.403 0.84 4.07 9.93 0.0036
CelebV-HQ test (150 clips) 51.5 248.8 0.574 0.77 4.76 10.1 0.0046

AURA-1.3B has the best identity preservation (CSIM) of all compared methods on both test sets. See the paper for the full tables.

License

AURA-1.3B starts from Wan2.1-T2V-1.3B: its teacher, AURA-1.3B-Teacher-TalkVerse, is Wan2.1-T2V-1.3B fine-tuned with LoRA and Mimi audio modules, and AURA-1.3B is distilled from that teacher. The weights (merged and LoRA-only) are released under CC BY-NC 4.0: non-commercial use only. This follows the terms of the training data: the student is distilled on teacher trajectories conditioned on reference images and prompts from the Hallo3 training data (CC BY-NC-ND 4.0), see the AURA dataset. The VAE and T5 text encoder you download from Wan2.1-T2V-1.3B keep their own license (Apache 2.0).

Responsible use

The model generates realistic videos of real-looking people speaking arbitrary audio. Do not use it to impersonate people, create non-consensual content or produce disinformation. Only animate portraits for which you have the subject's consent, and disclose that videos are AI-generated.

Contact

For questions, contact Nathaniel Cohen (nathaniel.cohen@kyutai.org) or AmΓ©lie Royer (amelie@kyutai.org).

Citation

@inproceedings{cohen2026aura,
  title     = {{AURA}: {AU}dio-d{R}iven streaming {A}vatar},
  author    = {Cohen, Nathaniel and Dufour, Nicolas and Royer, Am{\'e}lie
               and Newson, Alasdair and P{\'e}rez, Patrick},
  booktitle = {British Machine Vision Conference (BMVC)},
  year      = {2026},
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for kyutai/AURA-1.3B

Adapter
(1)
this model
Finetunes
1 model

Dataset used to train kyutai/AURA-1.3B

Collection including kyutai/AURA-1.3B

Paper for kyutai/AURA-1.3B