Instructions to use MC7ever/pe-graft-lfm with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use MC7ever/pe-graft-lfm with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "image-to-text" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("image-to-text", model="MC7ever/pe-graft-lfm")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("MC7ever/pe-graft-lfm", device_map="auto") - PerceptionEncoder
How to use MC7ever/pe-graft-lfm with PerceptionEncoder:
# Use any PE model as a vision encoder import core.vision_encoder.pe as pe model = pe.VisionTransformer.from_config("MC7ever/pe-graft-lfm", pretrained=True) - Notebooks
- Google Colab
- Kaggle
pe-graft-lfm β PE encoder grafted onto LFM2.5-2.6B (projector-only)
graft.pt (72MB) holds a trained 2-layer MLP projector (1024 -> 2048 x 4
prefix tokens) mapping frozen MC7ever/pe-single-L14 embeddings into frozen
LiquidAI/LFM2.5-2.6B input space. Both giants stay frozen; only the
projector trained (COCO-train 4000 pairs x 2 epochs, local Apple-Silicon MPS).
Run
git clone https://github.com/shadyuwugurl/pe-single-L14
python3 graft_generate.py --graft ./pe-graft-lfm --image photo.jpg
python3 graft_generate.py --graft ./pe-graft-lfm --audio clip.wav --sr 44100
The script downloads both base models automatically on first run.
Samples (honest)
- moped guy -> "A man riding a bike on the road." CORRECT
- Wii game ("two smiling young people playing Wii") -> "Two girls are in a park and one is taking pictures of the other." WRONG scene, right headcount
- restaurant meal -> "A coffee cup sits on a table with other items." gist-level
Limits
Projector-only with 4 prefix tokens: fluent, gist-level, not sharp. Path to sharp: more prefix tokens (4->16), LLM LoRA, audio prefix training, more data. Audio prefix path is wired but essentially untrained (v1 trained image->text only).