Add model card with usage instructions and technical details

README.md

@@ -1,3 +1,108 @@
----
-license: mit
----
+---
+license: mit
+library_name: diffusers
+pipeline_tag: image-to-image
+tags:
+  - image-inpainting
+  - object-removal
+  - object-insertion
+  - flux
+  - lora
+  - iccv2025
+base_model:
+  - black-forest-labs/FLUX.1-dev
+---
+
+# OmniPaint: Mastering Object-Oriented Editing via Disentangled Insertion-Removal Inpainting
+
+<p align="center">
+  <b>ICCV 2025</b>
+</p>
+
+<p align="center">
+  <a href="https://arxiv.org/pdf/2503.08677"><img src="https://img.shields.io/badge/Paper-arXiv-b31b1b.svg"></a>
+  <a href="https://www.yongshengyu.com/OmniPaint-Page/"><img src="https://img.shields.io/badge/Website-Project-6535a0"></a>
+  <a href="https://github.com/yeates/OmniPaint"><img src="https://img.shields.io/badge/GitHub-Code-181717?logo=github"></a>
+  <a href="https://huggingface.co/datasets/yeates/omnipaint-bench"><img src="https://img.shields.io/badge/HuggingFace-Dataset-FFD21E?logo=huggingface"></a>
+</p>
+
+## Model Description
+
+OmniPaint is a unified framework for **object-oriented image editing** that disentangles object insertion and removal into two specialized inpainting tasks. Built on top of [FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev), OmniPaint uses **LoRA adapters** to enable:
+
+- **Object Removal** — Remove foreground objects and their visual effects (shadows, reflections) using only object masks
+- **Object Insertion** — Seamlessly insert objects into existing scenes with harmonious lighting and perspective
+
+The model also introduces a novel **CFD (Content-aware Frechet Distance)** metric for reference-free evaluation of object removal quality.
+
+## Model Files
+
+| File | Description |
+|------|-------------|
+| `weights/omnipaint_remove.safetensors` | LoRA weights for object removal |
+| `weights/omnipaint_insert.safetensors` | LoRA weights for object insertion |
+| `embeddings/remove.npz` | Pre-computed text embeddings for removal |
+| `embeddings/insert.npz` | Pre-computed text embeddings for insertion |
+
+## Usage
+
+### Installation
+
+```bash
+git clone https://github.com/yeates/OmniPaint.git
+cd OmniPaint
+bash scripts/setup.sh
+```
+
+This will install dependencies and automatically download model weights from this repository.
+
+### Object Removal
+
+```bash
+python scripts/omnipaint_remove.py \
+  --input ./demo_assets/removal_samples/images/5.jpg \
+  --mask ./demo_assets/removal_samples/masks/5.png \
+  --output_dir ./outputs \
+  --seed 42 --steps 28 --device cuda:0
+```
+
+### Object Insertion
+
+```bash
+python scripts/omnipaint_insert.py \
+  --background ./demo_assets/insertion_samples/backgrounds/background-2.png \
+  --mask ./demo_assets/insertion_samples/masks/mask-2.png \
+  --subject ./demo_assets/insertion_samples/subjects/subject-2.png \
+  --output_dir ./outputs \
+  --seed 42 --steps 28 --device cuda:0
+```
+
+### Gradio Demo
+
+```bash
+bash scripts/app_setup.sh  # one-time setup (~15 min)
+python app.py
+```
+
+## Technical Details
+
+- **Base model**: [FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) (FluxTransformer2DModel)
+- **Training method**: LoRA fine-tuning with disentangled adapters for insertion and removal
+- **Framework**: Diffusers + PEFT
+- **Inference steps**: 28 (default)
+- **Guidance scale**: 3.5 (default)
+
+## Citation
+
+```bibtex
+@article{yu2025omnipaint,
+  title={OmniPaint: Mastering Object-Oriented Editing via Disentangled Insertion-Removal Inpainting},
+  author={Yu, Yongsheng},
+  journal={arXiv preprint arXiv:2503.08677},
+  year={2025}
+}
+```
+
+## License
+
+This model is released under the [MIT License](https://opensource.org/licenses/MIT).