TKCAM checkpoints

Pretrained checkpoints for TKCAM: Text and Keyframe to Camera Trajectory Generation (NeurIPS 2026). TKCAM generates a camera trajectory from a text description, optionally conditioned on a first frame or on up to 4 keyframes placed at chosen times.

Paper (arXiv:2610.11105) · Code · Project page · Dataset

mode	conditioning	mask transformer	residual transformer
noframe	text	mtrans_25kfixed_noframe_r4 (epoch 555)	rtrans_25kfixed_noframe_r4 (epoch 40)
firstframe	text + first frame	mtrans_25kfixed_1frame_r4 (epoch 480)	rtrans_25kfixed_1frame_r4 (epoch 600)
xframe	text + up to 4 keyframes	mtrans_25kfixed_xframe_r4 (epoch 600)	rtrans_25kfixed_xframe_r4 (epoch 125)

All modes share the residual VQ-VAE rvq_w128_nq4_cb256_25kfixed_60ep (4 quantizers, 256 codes, window 128; epoch 25). Text is encoded with a frozen pretrained t5-base and images with a frozen pretrained CLIP ViT-B/32 vision encoder (openai/clip-vit-base-patch32).

Usage

git clone https://github.com/linearalgebrayhz/TKCAM.git && cd TKCAM
pip install -r requirements.txt && pip install -e .     # or: conda env create -f environment.yml
hf download yhz-linearalgebra/TKCAM --local-dir checkpoints

# Text only
python scripts/generate.py --mode noframe --duration 6 \
  --text "The camera dollies forward slowly, revealing the living room."
# Text + first frame
python scripts/generate.py --mode firstframe --duration 5 --first_frame room.jpg \
  --text "The camera pans left smoothly across the kitchen."
# Text + keyframes at chosen times (image@seconds, up to 4)
python scripts/generate.py --mode xframe --duration 8 \
  --keyframe hall.jpg@0 --keyframe stairs.jpg@4.5 \
  --text "The camera moves forward through the hallway, then tilts up toward the stairs."

Each sample is saved as sample{i}_9d.npy ((T, 9): [rot6d(6), x, y, z]), sample{i}_c2w.npy ((T, 4, 4)), an MP4 animation and meta.json. Trajectories are camera-to-world, relative to the first frame, with OpenGL camera axes (+X right, +Y up, −Z forward), at 30 fps. Durations are clamped to 3–11 s; the training clips are about 4.0–9.3 s. See the code repository for batch inference over a prompts file and for training.

The T5 and CLIP encoder weights are not stored in these files: they are the unmodified public weights and are downloaded from the Hugging Face Hub on first use.

Files

The folder mirrors the layout the TKCAM code expects under checkpoints/:

realestate10k_rotmat/
  rvq_w128_nq4_cb256_25kfixed_60ep/{opt.txt, meta/mean.npy, meta/std.npy, model/net_best_recon.tar}
  {mtrans,rtrans}_25kfixed_{noframe,1frame,xframe}_r4/{opt.txt, net_best_fid.tar}
MANIFEST.json      # epochs, source checkpoint, parameter counts, sizes, sha256

Each .tar holds the model state_dict and the epoch (no optimizer state). Every transformer file is named net_best_fid.tar, the name the TKCAM scripts load by default: it is the checkpoint selected on validation FID (see Training); the epochs are in the table above and in MANIFEST.json. opt.txt is the training configuration; local paths in it were replaced with placeholders.

Training

• Data: the train split of RealEstate10K-Cap (21,250 clips), the released version of the dataset.
• Recipe: MoMask three-stage pipeline (residual VQ-VAE → masked transformer → residual transformer), 600 epochs per transformer, learning rate 5e-5. Visual conditioning uses the first frame (firstframe) or 0–4 randomly sampled keyframes per clip (xframe).
• Checkpoint selection: the VQ-VAE by validation reconstruction error. For each transformer, the saved checkpoints of the run (best validation FID during training, best accuracy, best loss, and the last epoch) were re-scored by FID on the 2,500-clip validation split with a CLaTr evaluator whose training data excludes the RealEstate10K-Cap validation and test clips; the best one is released as net_best_fid.tar.

Validation split (2,500 clips), CLaTr evaluator; R@k = text-to-trajectory retrieval over the whole split, in %:

mode	FID ↓	R@1	R@3	R@10	matching
noframe	0.042	2.1	5.2	13.5	0.289
firstframe	0.045	2.5	5.6	13.6	0.279
xframe	0.045	2.6	5.7	14.7	0.288

License

MIT, as the code. TKCAM builds on MoMask (MIT).

Citation

@misc{yang2026tkcam,
  title={TKCAM: Text and Keyframe to Camera Trajectory Generation},
  author={Yang, Haozhe and Dou, Zhiyang and Gu, Zekai and Lin, Cheng and Wang, Wenping and Liu, Yuan and Komura, Taku},
  year={2026},
  eprint={2610.11105},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2610.11105}
}