blanchon/ACE-Ego-Hand-Safetensors

ACE-Ego-Hand (safetensors)

Inference weights of ACE-Ego-Hand (Liu et al., 2026, arXiv:2608.20308), repackaged as safetensors for the reimplementation at https://github.com/julien-blanchon/ACE-Ego-Hand-Pipeline. The model recovers bimanual 3D hand motion (MANO pose, shape and camera-space translation per frame) from egocentric RGB video by reading the hidden state of a Wan2.2 video diffusion transformer.

Nothing here is retrained. Every tensor is a byte-for-byte copy of its source, renamed and, for the transformer, pruned to the blocks the model actually runs.

Files

path	what	source
dit/	Wan2.2-5B trunk, blocks 0-15 only (block 15 is the feature tap), bf16	alibaba-pai/Wan2.2-Fun-5B-Control
vae/	Wan2.2 VAE encoder (48 latent channels, 16x / 4x), bf16	Wan-AI/Wan2.2-TI2V-5B-Diffusers
taehv/	TAEHV taew2_2 encoder, the tiny approximation of the VAE, fp32	madebyollin/taehv
caption_embedding.safetensors	the one constant umT5-XXL caption embedding the trunk cross-attends to, fp32	computed from Wan-AI/Wan2.2-TI2V-5B-Diffusers
detector/	YOLOv8-m hand detector (26 M, fp32), the second opinion behind InferenceConfig.verify="detector"; CC-BY-NC-4.0 weights, see detector/README.md	rolpotamias/WiLoR detector.pt
kfree/, k/	per variant: backbone_delta.safetensors (LoRA rank 64 on blocks 0-15 + the trained patch embedding, fp32), projector/ (the readout: alternating encoder, heads, ray map), config.json	acerobotics2025/ACE-Ego-Hand
mano/	both MANO hands as tensors (template, blend shapes, regressor, skinning, faces), fp32	MANO
config.json	variants, encoders, encode width, window length

Each dit/, vae/, taehv/, mano/ and */projector/ folder is a config.json + model.safetensors pair loadable with huggingface_hub.PyTorchModelHubMixin (from_pretrained(folder)), and has a README saying what it is.

kfree needs no camera calibration (the model predicts a ray field and fits its own pinhole); k takes the intrinsics. Both are pinhole-only.

Usage

uv run --with "ace-ego-hand @ git+https://github.com/julien-blanchon/ACE-Ego-Hand-Pipeline" \
    ace-ego-hand infer https://huggingface.co/spaces/blanchon/ACE-Ego-Hand/resolve/main/examples/holoassist_coffee_machine.mp4 --render

That downloads the weights from this repository on first use, writes holoassist_coffee_machine.hands.parquet (one row per frame and hand) and the rendered holoassist_coffee_machine.hands.mp4 in the working directory. Any local file or URL works in place of the example.

See the repository README for the Python API, the LeRobot annotation script and the demo.

Variants

The int8 branch of this repository holds the same weights with the trunk (dit/) quantized by torchao to int8 weight-only: the 160 block linears keep int8 storage and a per-row scale and are dequantized to bf16 at the matmul, everything else (patch, time and text embeddings, norms, the LoRA adapters of the variant folders) is unchanged. It halves the trunk on disk (5.42 -> 2.80 GB) and cuts peak GPU memory by about 2.4 GiB at the same speed on CUDA, for a joint error of 0.45-0.8 mm median against the bf16 trunk (whose run-to-run noise floor is 0.35 mm). Only dit/ differs on that branch; the loader fetches it from there and every other component from main. Use it with --model.trunk-precision int8 (trunk_precision="int8" in ModelConfig), which needs the int8 dependency extra (ace-ego-hand[int8], i.e. torchao>=0.16); see dit/README.md on the branch for the file layout.

Licence

The ACE-Ego-Hand weights are CC BY-NC 4.0 (they derive from MANO and research-licensed datasets; no commercial use). The Wan2.2 weights are Apache 2.0. TAEHV is MIT. MANO is under its own licence from the Max Planck Institute.

Citation

@misc{liu2026aceegohand,
    title={ACE-Ego-Hand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery},
    author={Yufei Liu and Xixi Wang and Hao Li and Ganlong Zhao and Kaitong Cai and Chengkai Jin and Chunxiao Liu and Jianbo Liu and Siyuan Huang and Xingang Pan and Hongsheng Li},
    year={2026},
    eprint={2608.20308},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://arxiv.org/abs/2608.20308},
}