kevinqu7/loc3r-vlm-llava

Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models

Kevin Qu ^1,2* , Haozhe Qi ³ , Mihai Dusmanu ¹ , Mahdi Rad¹, Rui Wang¹, Marc Pollefeys^1,2

¹Microsoft Spatial AI Lab          ²ETH Zurich          ³EPFL

*work done during an internship at Microsoft

We introduce Loc3R-VLM, a novel framework that equips 2D VLMs with advanced 3D spatial understanding capabilities from video. Inspired by human cognition, it builds an internal cognitive map of the global environment while explicitly modeling an agent's position and orientation. By jointly capturing global layout and egocentric state, the model excels at two core tasks: language-driven localization and viewpoint-aware 3D reasoning.

This repository hosts the released model checkpoint, which uses LLaVA-Video-7B-Qwen2 as the base model.

More details can be found in our GitHub repository.

Citation

@misc{qu2026loc3rvlm,
    title={Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models},
    author={Kevin Qu and Haozhe Qi and Mihai Dusmanu and Mahdi Rad and Rui Wang and Marc Pollefeys},
    year={2026},
    eprint={2603.18002},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://arxiv.org/abs/2603.18002}
}