pe-graft-lfm β€” PE encoder grafted onto LFM2.5-2.6B (projector-only)

graft.pt (72MB) holds a trained 2-layer MLP projector (1024 -> 2048 x 4 prefix tokens) mapping frozen MC7ever/pe-single-L14 embeddings into frozen LiquidAI/LFM2.5-2.6B input space. Both giants stay frozen; only the projector trained (COCO-train 4000 pairs x 2 epochs, local Apple-Silicon MPS).

Run

git clone https://github.com/shadyuwugurl/pe-single-L14
python3 graft_generate.py --graft ./pe-graft-lfm --image photo.jpg
python3 graft_generate.py --graft ./pe-graft-lfm --audio clip.wav --sr 44100

The script downloads both base models automatically on first run.

Samples (honest)

  1. moped guy -> "A man riding a bike on the road." CORRECT
  2. Wii game ("two smiling young people playing Wii") -> "Two girls are in a park and one is taking pictures of the other." WRONG scene, right headcount
  3. restaurant meal -> "A coffee cup sits on a table with other items." gist-level

Limits

Projector-only with 4 prefix tokens: fluent, gist-level, not sharp. Path to sharp: more prefix tokens (4->16), LLM LoRA, audio prefix training, more data. Audio prefix path is wired but essentially untrained (v1 trained image->text only).

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support