qev_fast: image input on the fast CUDA engine; qev.py / qev_server.py images; card speed numbers

README.md

@@ -71,7 +71,7 @@ every move is one forward pass of about 4 ms.
 | memory | 1.7 GB weights in bf16, 0.8 GB in 8-bit · **1.3 GB RAM** on a Mac in 8-bit |
 | also first on | JevExtra (72.0%, agent and safety skills), JevSpatial (74.2%, games and space), Rules-7 (63.7%, tied) |
 | output | calibrated probabilities for `choice`, `noul` (yes / no) and `score` questions, up to 255 options each |
-| images | zero-shot image input with `qev_image.py` (`<image:NAME>` in the state, ≤196 tokens per image): 0.759 on JevVision v0 vs 0.651 for its base · [details](#images) |
+| images | zero-shot image input (`<image:NAME>` in the state, ≤196 tokens per image) on the fast CUDA engine or with `qev_image.py`: 0.759 on JevVision v0 vs 0.651 for its base · [details](#images) |
 | formats | PyTorch + fast CUDA engine (this repo), [MLX](https://huggingface.co/labqoat/qev-0.8b-MLX), [GGUF](https://huggingface.co/labqoat/qev-0.8b-GGUF) (also served by stock `llama-server -hf labqoat/qev-0.8b-GGUF:Q8_0` at `/v1/systemone`) |
 | license | Apache-2.0 |
 
@@ -275,11 +275,24 @@ MPS or CPU.
 
 ## Images
 
-qev can also read images. `qev_image.py` runs the same model with the vision tower it inherited from
-Qwen3.5-0.8B-Base: put `<image:NAME>` in the state where the image belongs and pass the image under that name. The
-prompt layout, the letter readout and the per-type temperatures are the same as in `qev.py`, and so is the answer
-format. A text-only request through `qev_image.py` gets the same answers as through `qev.py` (0 top-answer flips on 50
-requests, 107 questions).
+qev can also read images, with the vision tower it inherited from Qwen3.5-0.8B-Base: put `<image:NAME>` in the state
+where the image belongs and pass the image under that name. The prompt layout, the letter readout and the per-type
+temperatures are the same as for text, and so is the answer format. On an NVIDIA GPU, `qev.py`'s fast engine
+(`qev_fast/`) takes images directly; `qev_image.py` runs the same model on plain transformers (any device). A text-only
+request through `qev_image.py` gets the same answers as through `qev.py` (0 top-answer flips on 50 requests, 107
+questions).
+
+```python
+from qev import QEV   # the fast CUDA engine (requirements-cuda.txt); images need pillow and torchvision
+
+m = QEV.from_pretrained("labqoat/qev-0.8b", images=True)   # images=True: load the vision tower now (else on first use)
+m.decide("A customer sent this screenshot: <image:shot>",
+         {"error": {"type": "noul", "instructions": "Does the screenshot show an error dialog?"}},
+         images={"shot": "screenshot.png"})                  # a path or a PIL image
+m.decide_batch([(state, questions, images), (state2, questions2)])   # images are optional per request
+```
+
+On other devices (Apple MPS, CPU, CUDA without flash-linear-attention), use `qev_image.py`:
 
 ```python
 from qev_image import QEVImage   # qev_image.py in this repo (transformers; needs pillow and torchvision)
@@ -295,11 +308,13 @@ m.decide_batch([(state, questions, images), ...])    # images are optional per r
 
 An image the state does not mention is appended at the end of the state. Several `<image:...>` markers in one state
 work. Each image is resized (aspect kept) to at most `max_pixels`, 448×448 by default, which costs **at most 196
-tokens per image** (224×224: 49; 640×640: 400 with `max_pixels=640*640`). `decide_batch(..., usage=True)` reports the
-input and image tokens of each request.
+tokens per image** (224×224: 49; 640×640: 400 with `max_pixels=640*640`; `QEV.from_pretrained(..., max_pixels=...)`
+on the fast engine). `decide_batch(..., usage=True)` reports the input and image tokens of each request (on the fast
+engine, the image tokens are in `last_image_tokens`).
 
-**HTTP.** `qev_image_server.py` serves the same System One API as `qev_server.py` (it reuses its request validation
-and server), plus an `"images"` field: `{name: base64 PNG or JPEG}`. The answer adds `usage.image_tokens`.
+**HTTP.** On the fast engine, `qev_server.py` accepts an `"images"` field, `{name: base64 PNG or JPEG}`, and the
+answer adds `usage.image_tokens` (requests without images are unchanged). `qev_image_server.py` serves the same API
+on the transformers path:
 
 ```sh
 python qev_image_server.py --model labqoat/qev-0.8b --port 8018
@@ -309,9 +324,17 @@ curl http://127.0.0.1:8018/v1/systemone -H 'Content-Type: application/json' -d '
   "questions": {"error": {"type": "noul", "instructions": "Does the screenshot show an error dialog?"}}}'
 ```
 
-**Speed.** Images go through the plain transformers path; the fast CUDA engine in `qev_fast/` is text-only. On one
-L40S at batch 1 that path takes 45 ms for a short text request and about 61–69 ms with one image (224×224 to
-768×768). Over the 1,306 benchmark requests below, in batches of 24, it answered about 42 requests/s.
+**Speed.** On an NVIDIA GPU, `qev.py`'s fast engine runs images itself: the vision tower once per distinct image, the
+image embeddings in the packed forward with 3-D rotary positions, CUDA graphs for small calls. On one L40S at batch 1,
+a request with one image takes about 18–32 ms (224×224: 18 ms, 448×448: 20 ms, 768×768: 32 ms; median of 15), and
+5–18 ms when the same image comes again (image embeddings are cached). The plain transformers path (`qev_image.py`)
+takes about 61–69 ms. A short text request still takes 3.5 ms. Over the 1,306 benchmark requests below, in batches of
+24, the fast engine answered about 73 requests/s and transformers about 59 requests/s in the same run (42 in our
+earlier measurement). The first pass in a new process was slower, about 35 requests/s, because GPU kernels are
+compiled once for new batch shapes. The fast engine's answers match `qev_image.py` up to bf16 round-off. On
+JevVision v0 it scores 0.757 pooled accuracy and 0.115 ECE, against 0.758 and 0.113. The largest probability
+difference is 0.05, about the same as between `qev_image.py`'s own batch-1 and batch-24 answers. Text requests are
+unchanged, with the same probabilities bit for bit and the same speed.
 
 ### JevVision v0 (zero-shot)
 
@@ -561,7 +584,7 @@ use of the weights is for the user to assess.
   no images in training. Boards are read close to chance (connect-four 0.38, chess 0.60, tic-tac-toe 0.59), counting
   real objects is weak (CLEVR 0.40), reading a value off a chart is 0.63, and real-photo yes/no questions are
   0.71–0.87. JevVision's rendered images are clean; screenshots and documents from the wild are untested. Image
-  requests run only on the transformers path (`qev_image.py`), not on the fast engine, MLX or GGUF.
+  requests run on the fast CUDA engine (`qev.py`) and the transformers path (`qev_image.py`), not on MLX or GGUF.
 - **Inherited behaviour.** The model inherits biases of the base model and of its data, part of which was labelled
   by larger models. Do not use it as the only decision maker for consequential decisions about people (medical,
   legal, credit, employment).
@@ -574,15 +597,15 @@ use of the weights is for the user to assess.
 | `config.json`, `tokenizer.json`, `tokenizer_config.json`, `vocab.json`, `merges.txt` | the Qwen3.5 config and tokenizer |
 | `qev_config.json` | temperatures, the 255 option-letter tokens, the prompt and readout description |
 | `qev.py`, `requirements.txt` | the runner (backends: fast on NVIDIA GPUs, transformers elsewhere) |
-| `qev_server.py` | an HTTP server with the System One API (`POST /v1/systemone`) |
-| `qev_fast/`, `requirements-cuda.txt` | the fast CUDA engine `qev.py` uses by default on NVIDIA GPUs |
+| `qev_server.py` | an HTTP server with the System One API (`POST /v1/systemone`; `"images"` on the fast engine) |
+| `qev_fast/`, `requirements-cuda.txt` | the fast CUDA engine `qev.py` uses by default on NVIDIA GPUs (text and images) |
 | `qev_image.py` | the image runner (`QEVImage`: `<image:NAME>` in the state; transformers) |
 | `qev_image_server.py` | the System One HTTP server with an `"images"` field |
 | `preprocessor_config.json`, `video_preprocessor_config.json` | the base model's image processor configs, unchanged |
 | `SHA256SUMS` | checksums |
 
 The checkpoint keeps the base's multimodal config and its vision tower and projector, unchanged (bit-identical to
-Qwen3.5-0.8B-Base's). `qev.py` does not use them; `qev_image.py` does.
+Qwen3.5-0.8B-Base's). `qev.py`'s fast engine and `qev_image.py` use them for images; text requests do not.
 
 **Tokenizer fix in this revision.** The model was trained with the Qwen2 pre-tokenizer pattern, which splits words
 on letters only. The earlier `tokenizer.json` carried the newer Qwen3.5 pattern, which also groups combining marks.

SHA256SUMS

@@ -1,4 +1,4 @@
-d9c53ac320026041a2f082b6b8bf9c878dabf884c3c3c128d213126075e0c850  README.md
+9b2e8ee98eae68faabd10a78ca8930efa2ac735d12a42c1c410ee056742ce07a  README.md
 d9ab6f1698a46bbf04895d171fa0666816745e0d7209b9493d44d3df0caa1b5f  config.json
 aff0dc4549575e81bc423fc4d6ef963eb938f1a0330a141da15fa6211e2c5fe2  figures/accuracy_vs_size.png
 95abf758a17186f4a04ce6ae4ebbf499a4bc68c67c4673c00b856ab85ee22567  figures/footprint_speed.png
@@ -11,19 +11,20 @@ f069d8b61331c9e0d845f070954770c4c011c42defb996eb01f7ff559770c724  figures/header
 a9d356d7bdf1ef4949e3e748e95b8e10ad9d4e2e838eddc38a0a7b6b94d1db8d  merges.txt
 d33a719c0eedc34a530879e3840ddf0a0769b5060a987aed24e84e1170b8beb5  model.safetensors
 27225450ac9c6529872ee1924fcb0962ff5634834f817040f444118116f4e516  preprocessor_config.json
-9259f1c924126a47f2ebcab3be555daa4ad0b03ea73c5269e7bca565114d910c  qev.py
+080cba4669a1caa43e0f94ff5943c1bc00328770c96b1c394f2e8a6f64ea4798  qev.py
 5260608d19e1d02c7a59d7ee0351ab4a5d57cf40154132e3122eb9b147f6adb7  qev_config.json
 13de61af9869524aedf9920ea0a0cc0cae4b5f05eceb0d1383e415f3077e7d38  qev_fast/__init__.py
 42f113f0e51b4c94fdc60c753e7cde9bf3a45ec3a647cc1e545eaf9381a91052  qev_fast/encoder.py
 ed65a44a68aefcc6397068db52274433cd372a2c5fff6b1f7f901ef543429941  qev_fast/engine.py
-60c19edd37fdce2bec875d926d4b75e228948bfb7b997daee3ba8ea0c52269d9  qev_fast/graphs.py
-daeab2668863c8e6c4d262b5b7af7017c82184076d5c5318749642289da84d76  qev_fast/model.py
+6bf88c5ce2089fbd603eef82680f3ede24acd8a949887e00c37fdb7e775b0479  qev_fast/graphs.py
+0986f7be0a5049ffdb0ad49b55ab7d84d2f7916b0a4c79d0d1f82c6ce6b265c3  qev_fast/model.py
 e9a6ef607a94a11f659f4d57fa1588c8b99b5288535ed9fb7db0a144fd817c55  qev_fast/prefix.py
-b4f7f3ba357539c1625597c0c6cee7a2b0d2aa37277e5c89f015fc6dc9749ae5  qev_fast/runner.py
+c849a299772c7186b5b30feb9db777ef12673271e739051ce0968f6bbc741b27  qev_fast/runner.py
 9a7e7a159b4b6c9c1755ccbaf6c11b433bac78f8ec67474833e61821d77999ef  qev_fast/systemone.py
+53c44b0101b889464a78cf7b8c8bfe518765610f39478df1b7a3e8f680cb02e4  qev_fast/vision.py
 e8d061404d312ca2cfbec5db8d5adbacb9fd2888e20997833f8080ed317e4184  qev_image.py
 1cfc5fbdc1b4317bd2df8f2e965ee466f95a33af5db8c949d157f86c93996d4e  qev_image_server.py
-ae63ac8be0365edb19e677f3e72898c0ddf38df28acc8b8f85efdf2f44fe3ad4  qev_server.py
+01239024d6bce903ae16a8bda08a56b6ecbf43d630b9bd590fb1a73e9c11e54c  qev_server.py
 d7f47d96a76f395a93bd9c04c78e5193d7390c162c86b9ea8ce7010ec78b7ac7  requirements-cuda.txt
 b60d5907148b1d4c9191a02b3910c485852af951a529527ebc1b30089939500f  requirements.txt
 466f27fefa1f44a7bb82cb1b42163c761e01dc6f7f51fc4f3d0a499a5a9b4249  tokenizer.json

qev.py

@@ -8,6 +8,11 @@
               "urgent": {"type": "noul", "instructions": "Is this urgent?"}})
     m.decide_batch([(state, questions), ...])
 
+    # images (fast CUDA backend; pillow + torchvision): "<image:NAME>" in the state, or appended at the end
+    m.decide("Customer screenshot: <image:shot>", {"err": {"type": "noul", "instructions": "Is an error shown?"}},
+             images={"shot": "screenshot.png"})                       # PIL image or path
+    m.decide_batch([(state, questions, {"shot": img}), (state2, questions2)])
+
     python qev.py --model labqoat/qev-0.8b --state '{"ticket": "..."}' --questions '{"urgent": {"type": "noul", "instructions": "Is this urgent?"}}'
 
 How a request is answered (no text is generated):
@@ -24,8 +29,13 @@ Backends: "fast" (the default on CUDA when flash-linear-attention is installed:
 with CUDA graphs and a shared-prefix per-question path; pip install -r requirements-cuda.txt) or "transformers"
 (plain transformers forward; CPU, Apple MPS or CUDA). backend="auto" picks fast when it can.
 
+Images: the fast backend reads images as qev_image.QEVImage does (same resize to at most max_pixels, default 448*448
+= 196 image tokens, same placeholders and prompt layout, same readout and temperatures): the checkpoint's vision tower
+once per distinct image, its embeddings at the image placeholders, 3-D M-RoPE positions. The transformers backend of
+this file is text-only (use qev_image.py there).
+
 Requires: torch, transformers (Qwen3.5 support), decider-ai==1.9.0 (Apache-2.0; prompt layout and answer format);
-for the fast backend also flash-linear-attention (requirements-cuda.txt).
+for the fast backend also flash-linear-attention (requirements-cuda.txt); for images pillow and torchvision.
 """
 
 from __future__ import annotations
@@ -196,6 +206,7 @@ class QEV:
         self.fast = fast  # a qev_fast.FastQEV (backend "fast"), else the transformers forward below
         self.backend = "fast" if fast is not None else "transformers"
         self.temps = dict(config["temperatures"])
+        self.last_image_tokens: list = []  # per request of the last decide_batch / distributions_batch call
         if fast is not None:
             return
         letters = config.get("letter_token_ids") or letter_ids(tok)
@@ -206,11 +217,13 @@ class QEV:
     @classmethod
     def from_pretrained(cls, path: str = "labqoat/qev-0.8b", device: str | None = None, dtype=None,
                         config: str | dict | None = None, revision: str | None = None, backend: str = "auto",
-                        precapture: bool = True) -> "QEV":
+                        precapture: bool = True, images: bool = False, max_pixels: int = 448 * 448,
+                        min_pixels: int = 64 * 64) -> "QEV":
         """path: a local directory or a Hugging Face repo id. backend: "auto" (fast on CUDA with
         flash-linear-attention, else transformers), "fast" or "transformers". dtype (transformers backend): bf16 on
         CUDA / MPS, fp32 on CPU by default. precapture (fast): capture the common CUDA graphs now, the rest in the
-        background."""
+        background. images (fast): load the vision tower now (else on the first request with images) and capture
+        the image graphs with the others; max_pixels / min_pixels bound the resized image (as qev_image.py)."""
         import sys
 
         import torch
@@ -251,8 +264,11 @@ class QEV:
             from qev_fast import FastQEV
 
             fast = FastQEV(path)
+            fast.max_pixels, fast.min_pixels = max_pixels, min_pixels
+            if images:
+                fast.enable_images(max_pixels, min_pixels)
             if precapture:
-                fast.precapture(background=True)
+                fast.precapture(background=True, images=images)
             return cls(None, fast.tok, cfg, torch.device("cuda"), torch.bfloat16, fast)
         tok = use_training_pretokenizer(AutoTokenizer.from_pretrained(path))
         model = AutoModelForCausalLM.from_pretrained(path, dtype=dtype).to(dev).eval()
@@ -292,8 +308,11 @@ class QEV:
         return [encode(self.tok, state, questions, mo, ms)]
 
     def _fast_probs(self, requests: list, per_question: bool) -> list:
-        """Fast backend: [(state, questions)] -> per request [(row encoding, tempered probabilities)]."""
+        """Fast backend: [(state, questions[, images])] -> per request [(row encoding, tempered probabilities)]."""
         f = self.fast
+        if any(len(r) > 2 and r[2] for r in requests):
+            return self._fast_image_probs(requests, per_question)
+        requests = [tuple(r[:2]) for r in requests]
         if per_question:
             res = []
             for s, q in requests:
@@ -316,50 +335,91 @@ class QEV:
             encs.append(e)
         return [[(e, tempered_from_slot_probs(e, p, self.temps))] for e, p in zip(encs, f.slot_probs(encs), strict=True)]
 
+    def _fast_image_probs(self, requests: list, per_question: bool) -> list:
+        """Fast backend, a call with images: rows from FastQEV.encode_requests (image placeholders expanded)."""
+        f = self.fast
+        for r in requests:
+            if not isinstance(r[1], dict) or not r[1]:
+                raise ValueError("questions: a non-empty {id: question} object")
+        if per_question:
+            res = []
+            for r in requests:
+                s, q, imgs = r[0], r[1], (r[2] if len(r) > 2 else None)
+                rows = []
+                for e in f.encode_requests([(s, {k: v}, imgs) for k, v in q.items()] if len(q) > 1 else [(s, q, imgs)]):
+                    if isinstance(e, Exception):
+                        raise e
+                    rows.append(e)
+                probs = f.per_question_probs(rows) if len(rows) > 1 else f.slot_probs(rows)
+                res.append([(r_, tempered_from_slot_probs(r_, p, self.temps)) for r_, p in zip(rows, probs, strict=True)])
+            return res
+        encs = f.encode_requests(requests)
+        for e in encs:
+            if isinstance(e, Exception):
+                raise e
+        return [[(e, tempered_from_slot_probs(e, p, self.temps))] for e, p in zip(encs, f.slot_probs(encs), strict=True)]
+
+    def _check_images(self, requests: list):
+        if self.fast is None and any(len(r) > 2 and r[2] for r in requests):
+            raise ValueError("images need the fast CUDA backend of qev.py; on other devices use qev_image.py")
+
+    def _fast_results(self, requests: list, per_question: bool) -> list:
+        res = self._fast_probs(requests, per_question)
+        self.last_image_tokens = [sum(r.get("image_tokens", 0) for r, _ in rs) for rs in res]
+        return res
+
     def distributions_batch(self, requests: list, per_question: bool = False) -> list:
-        """[(state, questions)] -> per request {question id: {option name: probability}} (tempered, unrounded)."""
+        """[(state, questions[, images])] -> per request {question id: {option name: probability}} (tempered,
+        unrounded). images: {name: PIL image or path} (fast backend)."""
+        self._check_images(requests)
         if self.fast is not None:
             out = []
-            for rs in self._fast_probs(requests, per_question):
+            for rs in self._fast_results(requests, per_question):
                 d = {}
                 for r, p in rs:
                     d.update(flat(r, p))
                 out.append(d)
             return out
         rows, owner = [], []
-        for i, (s, q) in enumerate(requests):
+        for i, (s, q, *_) in enumerate(requests):
             for r in self._rows(s, q, per_question):
                 rows.append(r)
                 owner.append(i)
+        self.last_image_tokens = [0] * len(requests)
         res = [{} for _ in requests]
         for i, r, lg in zip(owner, rows, self._logits(rows), strict=True):
             res[i].update(flat(r, probabilities(r, lg, self.temps)))
         return res
 
     def decide_batch(self, requests: list, per_question: bool = False, usage: bool = False) -> list:
-        """[(state, questions)] -> per request {question id: answer} (decider's /v1/systemone answer format).
-        usage=True: per request (answers, input tokens read)."""
+        """[(state, questions[, images])] -> per request {question id: answer} (decider's /v1/systemone answer
+        format). images: {name: PIL image or path} (fast backend). usage=True: per request (answers, input tokens
+        read, image tokens included); the image tokens per request are in last_image_tokens."""
+        self._check_images(requests)
         if self.fast is not None:
             out = []
-            for rs in self._fast_probs(requests, per_question):
+            for rs in self._fast_results(requests, per_question):
                 d = {}
                 for r, p in rs:
                     d.update(answers(r, p))
                 out.append((d, sum(len(r["ids"]) for r, _ in rs)) if usage else d)
             return out
         rows, owner = [], []
-        for i, (s, q) in enumerate(requests):
+        for i, (s, q, *_) in enumerate(requests):
             for r in self._rows(s, q, per_question):
                 rows.append(r)
                 owner.append(i)
+        self.last_image_tokens = [0] * len(requests)
         res, toks = [{} for _ in requests], [0] * len(requests)
         for i, r, lg in zip(owner, rows, self._logits(rows), strict=True):
             res[i].update(answers(r, probabilities(r, lg, self.temps)))
             toks[i] += len(r["ids"])
         return list(zip(res, toks, strict=True)) if usage else res
 
-    def decide(self, state, questions: dict, per_question: bool = False) -> dict:
-        return self.decide_batch([(state, questions)], per_question)[0]
+    def decide(self, state, questions: dict, per_question: bool = False, images: dict | None = None) -> dict:
+        """images: {name: PIL image or path} (fast backend); "<image:NAME>" in the state places one, the others go at
+        the end of the context."""
+        return self.decide_batch([(state, questions, images) if images else (state, questions)], per_question)[0]
 
 
 def main():
@@ -371,8 +431,11 @@ def main():
     ap.add_argument("--per-question", action="store_true", help="one prompt row per question")
     ap.add_argument("--device", default=None)
     ap.add_argument("--backend", default="auto", choices=["auto", "fast", "transformers"])
+    ap.add_argument("--image", action="append", default=[], help="NAME=PATH (repeatable; fast backend)")
+    ap.add_argument("--max-pixels", type=int, default=448 * 448)
     a = ap.parse_args()
-    m = QEV.from_pretrained(a.model, device=a.device, backend=a.backend)
+    m = QEV.from_pretrained(a.model, device=a.device, backend=a.backend, max_pixels=a.max_pixels)
+    imgs = dict(x.split("=", 1) for x in a.image) or None
     if a.requests:
         reqs = [json.loads(line) for line in open(a.requests) if line.strip()]
         for r in m.decide_batch([(r["state"], r["questions"]) for r in reqs], a.per_question):
@@ -384,7 +447,8 @@ def main():
         state = json.loads(a.state)
     except json.JSONDecodeError:
         state = a.state
-    print(json.dumps(m.decide(state, json.loads(a.questions), a.per_question), indent=1, ensure_ascii=False))
+    print(json.dumps(m.decide(state, json.loads(a.questions), a.per_question, images=imgs), indent=1,
+                     ensure_ascii=False))
 
 
 if __name__ == "__main__":

qev_fast/graphs.py

@@ -17,6 +17,10 @@ captured on first use, replays the whole forward:
 - the letter logits at the answer slots, masked to each slot's options, softmax -> probabilities [rows * max_slots, L].
 
 Rows longer than the largest bucket, more rows or more slots than the limits: the caller's eager path.
+
+Image rows (encodings with "img_emb", "img_rows", "pos3"; qev_fast.runner) use their own graphs, keyed ("i", rows,
+length): the token embeddings of the image rows replaced from a static embedding buffer and the rotary tables built
+from static 3-D M-RoPE positions. The text graphs are not changed by them.
 """
 
 import time
@@ -60,6 +64,7 @@ class GraphSociety:
                   "uncertain": spec.get("k", 0)}[spec["kind"]] if self.E else 0
         self.graphs: dict = {}
         self.max_tokens = MAX_GRAPH_TOKENS
+        self.img_buf = None  # (embeddings [max_tokens, H], mask [max_tokens, 1], pos3 [3, max_tokens]): image graphs
         self.pool = torch.cuda.graph_pool_handle()
         self.stats = {"graph": 0, "captures": 0, "capture_s": 0.0, "capture_failed": []}
 
@@ -128,13 +133,18 @@ class GraphSociety:
     def _forward(self, st):
         fs, fm = self.fs, self.fs.fm
         n, T, b = st["n"], st["T"], st["b"]
-        rope = M.rope_tables(b.pos, fm.inv_freq, fm.embed.weight.dtype)
+        if "pos3" in st:  # image rows: 3-D positions
+            rope = M.mrope_tables(st["pos3"], fm.inv_freq, fm.embed.weight.dtype, fm.mrope_section)
+        else:
+            rope = M.rope_tables(b.pos, fm.inv_freq, fm.embed.weight.dtype)
         fm.m_tok = None
         fm.lora_gather = None
         if st.get("h6") is not None:  # the hidden state at router_layer given (prefix-shared per-question rows)
             h = st["h6"]
         else:
             h = fm.embed(b.ids)
+            if "img_mask" in st:
+                h = torch.where(st["img_mask"], st["img_emb"], h)
             for i in range(fs.router_layer):
                 h = fm.layer(i, h, b, rope)
         mask = st["mask"]  # [n, T] float32, 1 on real tokens
@@ -167,7 +177,15 @@ class GraphSociety:
                                     float("-inf"))
         return torch.softmax(logits, -1)
 
-    def _capture(self, n, T, upper: bool = False):
+    def _image_buffers(self):
+        if self.img_buf is None:
+            w = self.fs.fm.embed.weight
+            self.img_buf = (torch.zeros(self.max_tokens, w.shape[1], device=w.device, dtype=w.dtype),
+                            torch.zeros(self.max_tokens, 1, device=w.device, dtype=torch.bool),
+                            torch.zeros(3, self.max_tokens, device=w.device, dtype=torch.long))
+        return self.img_buf
+
+    def _capture(self, n, T, upper: bool = False, image: bool = False):
         t0 = time.perf_counter()
         b = Batch([[self.pad_id] * T for _ in range(n)], [[0] * self.max_slots for _ in range(n)], "cuda")
         if n > 1:  # equal-length rows as a real batch: no cu_seqlens (fla's varlen kernels are not graph-safe)
@@ -176,6 +194,9 @@ class GraphSociety:
         st = {"n": n, "T": T, "b": b, "mask": torch.ones(n, T, device=dev), "count": torch.full((n,), float(T), device=dev),
               "nopts": torch.ones(n * self.max_slots, dtype=torch.long, device=dev),
               "slot_row": torch.arange(n, device=dev).repeat_interleave(self.max_slots)}
+        if image:  # views of the shared image buffers (static addresses)
+            emb, msk, p3 = self._image_buffers()
+            st["img_emb"], st["img_mask"], st["pos3"] = emb[: n * T], msk[: n * T], p3[:, : n * T]
         if upper:
             st["h6"] = torch.zeros(n * T, self.fs.fm.embed.weight.shape[1], device=dev, dtype=self.fs.fm.embed.weight.dtype)
         s = torch.cuda.Stream()
@@ -215,6 +236,21 @@ class GraphSociety:
                 "gpu_reserved_gb": round(torch.cuda.memory_reserved() / 1e9, 2),
                 "failed": self.stats["capture_failed"][:3]}
 
+    def precapture_images(self, max_rows: int = 4, lengths=(192, 256, 320, 384, 448, 512, 640, 768, 1024),
+                          lock=None) -> dict:
+        """Capture the image graph buckets (rows <= max_rows, these length buckets)."""
+        import contextlib
+
+        t0 = time.perf_counter()
+        for T in lengths:
+            for n in self.row_buckets:
+                if n <= max_rows and n * T <= self.max_tokens and ("i", n, T) not in self.graphs:
+                    with lock if lock is not None else contextlib.nullcontext():
+                        self.graphs[("i", n, T)] = self._capture(n, T, image=True)
+        torch.cuda.synchronize()
+        return {"image_graphs": sum(1 for k, g in self.graphs.items() if k[0] == "i" and g is not None),
+                "s": round(time.perf_counter() - t0, 1)}
+
     def warm(self, shapes):
         for n, T in shapes:
             key = (self._bucket(n, self.row_buckets), self._bucket(T, self.len_buckets))
@@ -282,9 +318,10 @@ class GraphSociety:
         """Society-style encodings (eligible) -> per row its slot probabilities (list of [nopts] lists)."""
         n = self._bucket(len(encs), self.row_buckets)
         T = self._bucket(max(len(e[0]["ids"]) for e in encs), self.len_buckets)
-        key = (n, T)
+        image = any("pos3" in e[0] for e in encs)
+        key = ("i", n, T) if image else (n, T)
         if key not in self.graphs:
-            self.graphs[key] = self._capture(n, T)
+            self.graphs[key] = self._capture(n, T, image=image)
         hit = self.graphs[key]
         if hit is None:
             return None  # not capturable: the caller's eager path
@@ -310,6 +347,25 @@ class GraphSociety:
         st["mask"].copy_(mask, non_blocking=True)
         st["count"].copy_(count, non_blocking=True)
         st["nopts"].copy_(nopts.view(-1), non_blocking=True)
+        if image:
+            import numpy as np
+
+            pos3 = torch.zeros(3, n, T, dtype=torch.long)
+            rows, embs = [], []
+            for j, e in enumerate(encs):
+                it = e[0]
+                L = len(it["ids"])
+                if "pos3" in it:
+                    pos3[:, j, :L] = torch.from_numpy(it["pos3"])
+                    rows.append(j * T + it["img_rows"])
+                    embs.append(it["img_emb"])
+                else:
+                    pos3[:, j, :L] = torch.arange(L)
+            st["pos3"].copy_(pos3.view(3, -1), non_blocking=True)
+            idx = torch.from_numpy(np.concatenate(rows)).to(st["img_mask"].device, non_blocking=True)
+            st["img_mask"].zero_()
+            st["img_mask"].index_fill_(0, idx, True)
+            st["img_emb"].index_copy_(0, idx, torch.cat(embs).to(st["img_emb"].dtype))
         g.replay()
         self.stats["graph"] += 1
         p = out.view(n, S, -1)[: len(encs)].cpu()

qev_fast/model.py

@@ -56,6 +56,18 @@ def rope_tables(pos, inv_freq, dtype):
     return emb.cos().to(dtype), emb.sin().to(dtype)
 
 
+def mrope_tables(pos3, inv_freq, dtype, section):
+    """cos / sin [T, rot] of 3-D positions pos3 [3, T] (t, h, w): Qwen3.5's interleaved M-RoPE (frequency j takes the
+    h position for j = 1, 4, ... < 3 section[1], the w position for j = 2, 5, ... < 3 section[2], else t; the
+    transformers apply_interleaved_mrope). Equal rows give exactly rope_tables(pos3[0])."""
+    f = pos3.float()[:, :, None] * inv_freq.float()[None, None]  # [3, T, rot / 2]
+    out = f[0].clone()
+    out[:, 1: 3 * section[1]: 3] = f[1][:, 1: 3 * section[1]: 3]
+    out[:, 2: 3 * section[2]: 3] = f[2][:, 2: 3 * section[2]: 3]
+    emb = torch.cat([out, out], -1)
+    return emb.cos().to(dtype), emb.sin().to(dtype)
+
+
 def apply_rope(x, cos, sin):
     """x [T, H, D] with partial rotary on the first cos.shape[-1] dims (apply_rotary_pos_emb)."""
     r = cos.shape[-1]
@@ -113,7 +125,9 @@ def attend(q, k, v, cu, cu_list, max_len: int, scale: float, backend: str):
 # ---- the model -------------------------------------------------------------------------------------------------------
 class Batch:
     """A packed forward's inputs on the device: ids [T], cu [N+1] int32 (None for one sequence), pos [T], slots [S]
-    (packed row of every answer slot), cu_cpu (host copy of cu), max_len."""
+    (packed row of every answer slot), cu_cpu (host copy of cu), max_len. Image rows (optional attributes, set by the
+    caller): img = (rows [R] long, embeddings [R, H]) replaces the token embeddings of those rows; pos3 [3, T] are the
+    M-RoPE positions (text-only batches have neither and run exactly as before)."""
 
     def __init__(self, seqs: list, slots: list, device, pad_to: int | None = None, pad_id: int = 0):
         """seqs: token id lists; slots: per sequence, slot offsets in it. pad_to: one sequence right-padded to this
@@ -163,6 +177,7 @@ class FastQwen35(nn.Module):
         self.register_buffer("letters", torch.as_tensor(letters, dtype=torch.long, device=self.embed.weight.device),
                              persistent=False)
         self.inv_freq = core.rotary_emb.inv_freq
+        self.mrope_section = list(getattr(core.rotary_emb, "mrope_section", None) or [11, 11, 10])
         self.L = []
         for i, layer in enumerate(core.layers):
             d = {"in_norm": layer.input_layernorm.weight, "post_norm": layer.post_attention_layernorm.weight,
@@ -280,7 +295,12 @@ class FastQwen35(nn.Module):
         """-> final-norm hidden states at the slots [S, H]. ckpt: "none", "full" (recompute every layer in backward)
         or "attn" (recompute only the full-attention layers: the delta-net layers keep their activations)."""
         h = self.embed(b.ids)
-        rope = rope_tables(b.pos, self.inv_freq, h.dtype)
+        img = getattr(b, "img", None)
+        if img is not None:  # image rows: the vision tower's embeddings instead of the placeholder token's
+            h = h.index_copy(0, img[0], img[1].to(h.dtype))
+        pos3 = getattr(b, "pos3", None)
+        rope = (rope_tables(b.pos, self.inv_freq, h.dtype) if pos3 is None
+                else mrope_tables(pos3, self.inv_freq, h.dtype, self.mrope_section))
         n = len(self.L)
         for i in range(n):
             rows = b.slots if i == n - 1 else None

qev_fast/runner.py

@@ -7,6 +7,12 @@
   state;
 - precapture(): the common graph buckets now, the rest in a background thread under the GPU lock.
 
+- images (enable_images(), qev_fast.vision): "<image:NAME>" placeholders expanded as qev_image.QEVImage does, the
+  checkpoint's vision tower run once per distinct image of a call, its embeddings scattered into the image rows and
+  3-D M-RoPE positions; image rows run on their own CUDA graphs (a separate graph family with an embedding buffer, so
+  the text graphs and the text path are unchanged) or packed eagerly; per_question rows with images run as an
+  ordinary batch (no prefix sharing).
+
 Outputs are each slot's option probabilities (softmax over its options, untempered); qev.py renormalises over the
 named options, tempers per type and formats the answers.
 """
@@ -45,6 +51,8 @@ def _softmax(xs):
 class FastQEV:
     def __init__(self, path: str, attn: str | None = None, graphs: bool = True, max_batch_tokens: int = 32768):
         t = time.perf_counter()
+        self.path = path
+        self.vision = None  # enable_images() loads the vision tower
         torch.backends.cuda.enable_cudnn_sdp(False)
         lm, self.tok, letters = E.load_a1(path, None)
         self.attn = attn or _attn_backend()
@@ -60,15 +68,18 @@ class FastQEV:
         self.load_s = time.perf_counter() - t
 
     # ---- warm-up ---------------------------------------------------------------------------------------------------
-    def precapture(self, background: bool = True, per_question: bool = False) -> dict:
+    def precapture(self, background: bool = True, per_question: bool = False, images: bool = False) -> dict:
         """Capture the common graph buckets (<= 4 rows, <= 2048 tokens) now; the rest in a background thread (or now,
-        background=False). per_question: the prefix / suffix graphs too."""
+        background=False). per_question: the prefix / suffix graphs too. images: the image graphs (<= 4 rows, 192 to
+        1024 tokens) too (after enable_images())."""
         if self.graphs is None:
             return {}
         with self.lock:
             info = self.graphs.precapture(max_rows=4, max_len=2048)
 
         def rest():
+            if images and self.vision is not None:
+                self.graphs.precapture_images(lock=self.lock)
             self.graphs.precapture(lock=self.lock)
             if per_question:
                 with self.lock:
@@ -80,9 +91,134 @@ class FastQEV:
             rest()
         return info
 
+    # ---- images -------------------------------------------------------------------------------------------------
+    def enable_images(self, max_pixels: int = 448 * 448, min_pixels: int = 64 * 64):
+        """Load the vision tower (once; needs pillow, torchvision and the checkpoint's preprocessor_config.json)."""
+        if self.vision is None:
+            from .vision import Vision
+
+            with self.lock:
+                self.vision = Vision(self.path)
+            self.marks = self.vision.tokens(self.tok)
+        self.max_pixels, self.min_pixels = max_pixels, min_pixels
+        return self
+
+    def encode_requests(self, requests: list) -> list:
+        """[(state, questions[, images])] -> per request its prompt row (encoder dict; with images also "images":
+        the Prepped images in placeholder order, "image_tokens") or an exception instance."""
+        from decider.systemone import render_state
+
+        plain, pre, errs = [], {}, {}
+        for i, r in enumerate(requests):
+            s, q, *rest = r
+            imgs = rest[0] if rest else None
+            if imgs:
+                if self.vision is None:
+                    self.enable_images()
+                try:
+                    cache: dict = {}
+
+                    def prepped_of(k, imgs=imgs, cache=cache):
+                        if k not in cache:
+                            cache[k] = self.vision.prep(imgs[k], self.min_pixels, self.max_pixels)
+                        return cache[k]
+
+                    from .vision import expand
+
+                    ctx, preps = expand(render_state(s), dict(imgs), prepped_of, self.marks)
+                    pre[i] = preps
+                    plain.append((ctx, q))
+                except (ValueError, KeyError, TypeError, OSError) as e:
+                    errs[i] = e
+                    plain.append((s, q))
+            else:
+                plain.append((s, q))
+        encs = self.encoder.encode_many(plain)
+        for i, e in errs.items():
+            encs[i] = e
+        for i, preps in pre.items():
+            e = encs[i]
+            if isinstance(e, Exception):
+                continue
+            want = sum(p.n for p in preps)
+            got = sum(1 for t in e["ids"] if t == self.vision.image_token_id)
+            if want != got:
+                encs[i] = RuntimeError(f"image placeholder tokens: expected {want}, the tokenizer produced {got}")
+                continue
+            e["images"], e["image_tokens"] = preps, want
+        return encs
+
+    def _attach_images(self, encs: list):
+        """Image rows: their embeddings (one tower forward for the call), image row offsets and M-RoPE positions."""
+        import numpy as np
+
+        embs = self.vision.embed([p for e in encs for p in e["images"]])
+        k = 0
+        for e in encs:
+            n = len(e["images"])
+            e["img_emb"] = torch.cat(embs[k: k + n]) if n > 1 else embs[k]
+            k += n
+            e["img_rows"] = np.nonzero(np.asarray(e["ids"]) == self.vision.image_token_id)[0]
+            e["pos3"] = self.vision.positions(e["ids"], e["images"])
+
+    def _image_probs(self, encs: list) -> list:
+        import numpy as np
+
+        out = [None] * len(encs)
+        with self.lock, torch.no_grad():
+            self._attach_images(encs)
+            if self.graphs is not None and len(encs) <= self.graphs.row_buckets[-1]:
+                for i, r in enumerate(self.graphs.probs_many([(e,) for e in encs])):
+                    if r is not None:
+                        out[i] = r
+            order = sorted((i for i in range(len(encs)) if out[i] is None), key=lambda i: len(encs[i]["ids"]))
+            group, tokens = [], 0
+
+            def flush():
+                from .model import Batch
+
+                b = Batch([encs[i]["ids"] for i in group], [encs[i]["slots"] for i in group], "cuda")
+                starts = np.concatenate([[0], np.cumsum([len(encs[i]["ids"]) for i in group])])
+                rows = np.concatenate([starts[j] + encs[i]["img_rows"] for j, i in enumerate(group)])
+                b.img = (torch.from_numpy(rows).to("cuda", non_blocking=True),
+                         torch.cat([encs[i]["img_emb"] for i in group]))
+                b.pos3 = torch.from_numpy(np.concatenate([encs[i]["pos3"] for i in group], 1)).to("cuda",
+                                                                                                 non_blocking=True)
+                lg = self.model(b).cpu()
+                k = 0
+                for i in group:
+                    ns = len(encs[i]["slots"])
+                    rows_ = lg[k: k + ns].float().tolist()
+                    k += ns
+                    out[i] = [_softmax(rows_[s][: encs[i]["nopts"][s]]) for s in range(ns)]
+
+            for i in order:
+                n = len(encs[i]["ids"])
+                if group and tokens + n > self.max_batch_tokens:
+                    flush()
+                    group, tokens = [], 0
+                group.append(i)
+                tokens += n
+            if group:
+                flush()
+        for e in encs:  # drop the device tensors (the vision cache keeps the embeddings)
+            e.pop("img_emb", None)
+        return out
+
     # ---- slot probabilities -------------------------------------------------------------------------------------------
     def slot_probs(self, encs: list) -> list:
-        """Prompt rows (encoder dicts) -> per row, per slot: softmax over the slot's options (untempered)."""
+        """Prompt rows (encoder dicts) -> per row, per slot: softmax over the slot's options (untempered). Rows with
+        images ("images" from encode_requests) run on the image path, the others exactly as text-only calls."""
+        if any(e.get("images") for e in encs):
+            img = [i for i, e in enumerate(encs) if e.get("images")]
+            txt = [i for i, e in enumerate(encs) if not e.get("images")]
+            out = [None] * len(encs)
+            for i, r in zip(img, self._image_probs([encs[i] for i in img]), strict=True):
+                out[i] = r
+            if txt:
+                for i, r in zip(txt, self.slot_probs([encs[i] for i in txt]), strict=True):
+                    out[i] = r
+            return out
         out = [None] * len(encs)
         with self.lock, torch.no_grad():
             if encs and self.graphs is not None and len(encs) <= self.graphs.row_buckets[-1]:
@@ -112,7 +248,7 @@ class FastQEV:
     def per_question_probs(self, rows: list) -> list:
         """The per-question rows of one request: their longest common prefix computed once (exact up to bf16
         round-off), else as an ordinary batch."""
-        if len(rows) > 1:
+        if len(rows) > 1 and not any(e.get("images") for e in rows):
             m = min(len(e["ids"]) for e in rows)
             first = rows[0]["ids"]
             lcp = 0

qev_fast/vision.py

@@ -0,0 +1,168 @@
+"""Image input for the fast engine: the same images, prompt layout and positions as qev_image.QEVImage.
+
+- `fit_size` / `prep`: an image (PIL or path) resized with aspect kept to at most max_pixels (default 448*448, 196
+  image tokens) and at least min_pixels, each side a multiple of 32, then the checkpoint's own image processor
+  (preprocessor_config.json, do_resize=False) -> pixel patches, grid (t, h, w) and N = h * w / 4 image tokens.
+- `expand`: "<image:NAME>" markers in the rendered state replaced by <|vision_start|> + <|image_pad|> x N +
+  <|vision_end|>; an image the state does not mention is appended at the end of the context.
+- `Vision`: the checkpoint's vision tower + merger (the `model.visual.*` tensors, the base model's) on the GPU, run
+  once per distinct image of a call (all images of the call packed in one tower forward), with a small cache keyed by
+  the image's pixel content.
+- `positions`: the 3-D M-RoPE positions of a prompt row (transformers' Qwen3.5 get_rope_index): text tokens
+  advance t = h = w by one; an image of llm grid (gh, gw) starting at p takes t = p, h = p + row, w = p + column and
+  the text after it continues at p + max(gh, gw).
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import math
+import os
+import re
+from collections import OrderedDict
+
+import numpy as np
+import torch
+
+IMAGE_MARK = re.compile(r"<image:([^<>\s]+)>")
+FACTOR = 32  # patch 16 x merge 2
+
+
+def fit_size(w: int, h: int, min_pixels: int, max_pixels: int, factor: int = FACTOR) -> tuple[int, int]:
+    """(width, height) a multiple of `factor` each, aspect kept, min_pixels <= w*h <= max_pixels (qev_image.fit_size)."""
+    hh = max(factor, round(h / factor) * factor)
+    ww = max(factor, round(w / factor) * factor)
+    if hh * ww > max_pixels:
+        beta = math.sqrt(h * w / max_pixels)
+        hh, ww = max(factor, math.floor(h / beta / factor) * factor), max(factor, math.floor(w / beta / factor) * factor)
+    elif hh * ww < min_pixels:
+        beta = math.sqrt(min_pixels / (h * w))
+        hh, ww = math.ceil(h * beta / factor) * factor, math.ceil(w * beta / factor) * factor
+    return ww, hh
+
+
+def _open(im):
+    from PIL import Image
+
+    if isinstance(im, (str, os.PathLike)):
+        im = Image.open(im)
+    return im.convert("RGB")
+
+
+class Prepped:
+    __slots__ = ("key", "n", "pix", "thw")
+
+    def __init__(self, pix, thw, n, key):
+        self.pix, self.thw, self.n, self.key = pix, thw, n, key
+
+
+class Vision:
+    """The vision tower + merger of a Qwen3.5 image-text checkpoint, and the image side of the prompt."""
+
+    def __init__(self, path: str, device="cuda", dtype=torch.bfloat16, cache_size: int = 256):
+        from transformers import AutoImageProcessor, AutoModelForImageTextToText
+
+        cfg = json.load(open(os.path.join(path, "config.json")))
+        self.image_token_id = cfg["image_token_id"]
+        self.start_id, self.end_id = cfg["vision_start_token_id"], cfg["vision_end_token_id"]
+        self.merge = cfg.get("vision_config", {}).get("spatial_merge_size", 2)
+        self.ip = AutoImageProcessor.from_pretrained(path)
+        # the full image-text model as qev_image loads it (same dtype handling); only its vision tower is kept
+        vlm = AutoModelForImageTextToText.from_pretrained(path, dtype=dtype)
+        self.visual = vlm.model.visual.to(device).eval()
+        for p in self.visual.parameters():
+            p.requires_grad_(False)
+        del vlm
+        self.device, self.dtype = torch.device(device), dtype
+        self.cache: OrderedDict = OrderedDict()
+        self.cache_size = cache_size
+        self.stats = {"images": 0, "cache_hits": 0, "tower_calls": 0}
+
+    def tokens(self, tok) -> tuple[str, str, str]:
+        c = tok.convert_ids_to_tokens
+        return c(self.start_id), c(self.image_token_id), c(self.end_id)
+
+    def prep(self, im, min_pixels: int, max_pixels: int) -> Prepped:
+        """PIL / path -> pixel patches [P, D] (CPU), grid [1, 3], image tokens, content key."""
+        im = _open(im)
+        w, h = fit_size(im.size[0], im.size[1], min_pixels, max_pixels)
+        im = im.resize((w, h), 3)
+        key = hashlib.blake2b(im.tobytes() + f"{w}x{h}".encode(), digest_size=16).hexdigest()
+        out = self.ip(images=[im], do_resize=False, return_tensors="pt")
+        thw = out["image_grid_thw"]
+        pix = out["pixel_values"]
+        return Prepped(pix, thw, int(thw.prod(-1).sum()) // (self.merge ** 2), key)
+
+    @torch.no_grad()
+    def embed(self, preps: list) -> list:
+        """Prepped images -> their merged embeddings [N_i, H] on the device (distinct uncached images in one tower
+        forward, packed by grid)."""
+        todo, seen = [], set()
+        for p in preps:
+            if p.key not in self.cache and p.key not in seen:
+                todo.append(p)
+                seen.add(p.key)
+        self.stats["images"] += len(preps)
+        self.stats["cache_hits"] += len(preps) - len(todo)
+        if todo:
+            pix = torch.cat([p.pix for p in todo]).to(self.device, self.visual.dtype)
+            thw = torch.cat([p.thw for p in todo]).to(self.device)
+            out = self.visual(pix, grid_thw=thw, return_dict=True).pooler_output
+            self.stats["tower_calls"] += 1
+            for p, e in zip(todo, torch.split(out, [p.n for p in todo]), strict=True):
+                self.cache[p.key] = e
+            while len(self.cache) > self.cache_size:
+                self.cache.popitem(last=False)
+        res = []
+        for p in preps:
+            self.cache.move_to_end(p.key)
+            res.append(self.cache[p.key])
+        return res
+
+    def positions(self, ids: list, preps: list) -> np.ndarray:
+        """[3, L] M-RoPE positions of one prompt row whose image runs are `preps` in order."""
+        ids = np.asarray(ids)
+        pos = np.empty((3, len(ids)), np.int64)
+        is_img = ids == self.image_token_id
+        cur, i, k, L = 0, 0, 0, len(ids)
+        while i < L:
+            if not is_img[i]:
+                j = i
+                while j < L and not is_img[j]:
+                    j += 1
+                pos[:, i:j] = np.arange(cur, cur + j - i)[None]
+                cur += j - i
+                i = j
+                continue
+            p = preps[k]
+            k += 1
+            _, gh, gw = (int(x) for x in p.thw[0].tolist())
+            gh, gw = gh // self.merge, gw // self.merge
+            assert gh * gw == p.n and bool(is_img[i: i + p.n].all()), "image placeholder run does not match its image"
+            hh, ww = np.meshgrid(np.arange(gh), np.arange(gw), indexing="ij")
+            pos[0, i: i + p.n] = cur
+            pos[1, i: i + p.n] = cur + hh.reshape(-1)
+            pos[2, i: i + p.n] = cur + ww.reshape(-1)
+            cur += max(gh, gw)
+            i += p.n
+        assert k == len(preps), "more images than placeholder runs"
+        return pos
+
+
+def expand(ctx: str, images: dict, prepped_of, marks: tuple[str, str, str]) -> tuple[str, list]:
+    """The rendered state with every image's placeholder run (qev_image.QEVImage._row) -> (context string, the
+    Prepped images in occurrence order). prepped_of(name) -> Prepped."""
+    named = []
+    for m in IMAGE_MARK.finditer(ctx):
+        if m.group(1) not in images:
+            raise ValueError(f"the state mentions <image:{m.group(1)}> but no such image was given")
+        named.append(m.group(1))
+    for k in images:  # an image the state does not mention goes at the end
+        if k not in named:
+            ctx += f"\n<image:{k}>"
+            named.append(k)
+    prepped = {k: prepped_of(k) for k in dict.fromkeys(named)}
+    s, pad, e = marks
+    ctx = IMAGE_MARK.sub(lambda m: s + pad * prepped[m.group(1)].n + e, ctx)
+    return ctx, [prepped[k] for k in named]

qev_server.py

@@ -14,8 +14,11 @@
 
 Endpoints:
 - POST /v1/systemone  {"state": str | object | array, "questions": {id: question}, "model": optional, "encoding":
-                       optional "joint" | "per_question"} -> {"model", "answers", "usage"}. Code written for the
-                       System One API works unchanged; "model" in the request is accepted and ignored.
+                       optional "joint" | "per_question", "images": optional {name: base64 PNG / JPEG}} ->
+                       {"model", "answers", "usage"}. Code written for the System One API works unchanged; "model" in
+                       the request is accepted and ignored. Images (fast CUDA backend; pillow + torchvision): the
+                       state places one with "<image:name>", an image it does not mention goes at the end of the
+                       context; usage then also reports "image_tokens".
 - GET  /health        readiness, backend, encoding, model files' SHA-256 (from SHA256SUMS when present).
 - GET  /v1/models     the served model.
 
@@ -49,14 +52,14 @@ class Batcher:
         self.q: queue.Queue = queue.Queue()
         threading.Thread(target=self._loop, daemon=True).start()
 
-    def submit(self, state, questions: dict, per_question: bool, timeout: float = 300.0):
+    def submit(self, state, questions: dict, per_question: bool, timeout: float = 300.0, images: dict | None = None):
         done, box = threading.Event(), {}
-        self.q.put((state, questions, per_question, done, box))
+        self.q.put((state, questions, per_question, done, box, images))
         if not done.wait(timeout):
             raise TimeoutError("the model did not answer in time")
         if "error" in box:
             raise box["error"]
-        return box["answers"], box["tokens"]
+        return box["answers"], box["tokens"], box.get("image_tokens", 0)
 
     def _loop(self):
         while True:
@@ -73,7 +76,7 @@ class Batcher:
 
     def _run(self, jobs: list, per_question: bool):
         try:
-            res = self.decide_batch([(j[0], j[1]) for j in jobs], per_question)
+            res = self.decide_batch([(j[0], j[1], j[5]) if j[5] else (j[0], j[1]) for j in jobs], per_question)
         except Exception as e:  # noqa: BLE001 - one bad request must not fail the others: answer each on its own
             if len(jobs) > 1:
                 for j in jobs:
@@ -82,8 +85,9 @@ class Batcher:
                 jobs[0][4]["error"] = e
                 jobs[0][3].set()
             return
-        for j, (ans, toks) in zip(jobs, res, strict=True):
-            j[4]["answers"], j[4]["tokens"] = ans, toks
+        for j, r in zip(jobs, res, strict=True):
+            j[4]["answers"], j[4]["tokens"] = r[0], r[1]
+            j[4]["image_tokens"] = r[2] if len(r) > 2 else 0
             j[3].set()
 
 
@@ -108,6 +112,33 @@ def _validate(req) -> tuple:
     return req["state"], qs, enc
 
 
+def decode_images(raw) -> dict | None:
+    """The "images" field: {name: base64 string (a data: URL prefix is accepted)} -> {name: PIL image}."""
+    import base64
+    import binascii
+    import io
+
+    if raw is None or raw == {}:
+        return None
+    if not isinstance(raw, dict):
+        raise ValueError('"images" must be an object {name: base64 string}')
+    try:
+        from PIL import Image
+    except ImportError:
+        raise ValueError("images need pillow (pip install pillow torchvision)") from None
+    out = {}
+    for k, v in raw.items():
+        if not isinstance(v, str):
+            raise ValueError(f'image "{k}": a base64 string')
+        if v.startswith("data:"):
+            v = v.split(",", 1)[-1]
+        try:
+            out[k] = Image.open(io.BytesIO(base64.b64decode(v, validate=True))).convert("RGB")
+        except (binascii.Error, OSError) as e:
+            raise ValueError(f'image "{k}": not a decodable base64 image ({e})') from None
+    return out
+
+
 def _file_hashes(path: str | None) -> dict:
     if path and os.path.exists(os.path.join(path, "SHA256SUMS")):
         out = {}
@@ -167,20 +198,26 @@ def make_handler(batcher: Batcher, info: dict, encoding: str, api_key: str | Non
                 n = int(self.headers.get("Content-Length", 0))
                 if n > MAX_BODY:
                     return self._send(413, {"detail": f"the body is larger than {MAX_BODY} bytes"})
-                state, qs, enc = _validate(json.loads(self.rfile.read(n) or b"{}"))
+                body = json.loads(self.rfile.read(n) or b"{}")
+                state, qs, enc = _validate(body)
+                imgs = decode_images(body.get("images"))
+                if imgs and not info.get("images"):
+                    raise ValueError("this server does not read images (the fast CUDA backend is needed)")
             except (ValueError, json.JSONDecodeError) as e:
                 return self._send(422, {"detail": str(e)})
             t = time.perf_counter()
             try:
-                ans, toks = batcher.submit(state, qs, (enc or encoding) == "per_question")
+                ans, toks, itoks = batcher.submit(state, qs, (enc or encoding) == "per_question", images=imgs)
             except (ValueError, KeyError, TypeError, AssertionError) as e:
                 return self._send(422, {"detail": str(e) or type(e).__name__})
             except TimeoutError as e:
                 return self._send(503, {"detail": str(e)})
             except Exception as e:  # noqa: BLE001
                 return self._send(500, {"detail": f"{type(e).__name__}: {e}"})
-            return self._send(200, {"model": info["model"], "answers": ans,
-                                    "usage": {"input_tokens": toks, "output_tokens": 0},
+            usage = {"input_tokens": toks, "output_tokens": 0}
+            if imgs:
+                usage["image_tokens"] = itoks
+            return self._send(200, {"model": info["model"], "answers": ans, "usage": usage,
                                     "latency_ms": round((time.perf_counter() - t) * 1000, 2)})
 
     return H
@@ -201,17 +238,27 @@ def main():
     ap.add_argument("--fast-start", action="store_true",
                     help="fast backend: listen once the common CUDA graphs are captured and capture the rest in the "
                          "background (requests during that time can be slow); default: capture all before listening")
+    ap.add_argument("--no-images", action="store_true",
+                    help="do not load the vision tower (requests with \"images\" then get HTTP 422)")
+    ap.add_argument("--max-pixels", type=int, default=448 * 448, help="images are resized to at most this many pixels")
     ap.add_argument("--verbose", action="store_true", help="log every request")
     a = ap.parse_args()
 
     from qev import QEV
 
     t0 = time.perf_counter()
-    m = QEV.from_pretrained(a.model, device=a.device, backend=a.backend, precapture=False)
+    m = QEV.from_pretrained(a.model, device=a.device, backend=a.backend, precapture=False, max_pixels=a.max_pixels)
+    images = False
+    if m.fast is not None and not a.no_images:
+        try:
+            m.fast.enable_images(a.max_pixels)
+            images = True
+        except Exception as e:  # noqa: BLE001 - no pillow / torchvision / processor config: a text-only server
+            print(f"qev_server: images off ({type(e).__name__}: {e})", flush=True)
     if m.fast is not None:  # CUDA graphs for every batch shape, so no live request waits for a capture
         print("qev_server: capturing CUDA graphs (about 2 min the first time, faster with a warm kernel cache)",
               flush=True)
-        m.fast.precapture(background=a.fast_start, per_question=a.encoding == "per_question")
+        m.fast.precapture(background=a.fast_start, per_question=a.encoding == "per_question", images=images)
     m.decide("warm-up", {"q": {"type": "noul", "instructions": "Is this a warm-up?"}})
     local = a.model if os.path.isdir(a.model) else None
     if local is None:
@@ -222,9 +269,13 @@ def main():
         except Exception:  # noqa: BLE001
             local = None
     info = {"model": "qev-0.8b", "source": a.model, "backend": m.backend, "device": str(m.device),
-            "encoding": a.encoding, "load_s": round(time.perf_counter() - t0, 1), "sha256": _file_hashes(local),
+            "encoding": a.encoding, "images": images, "load_s": round(time.perf_counter() - t0, 1), "sha256": _file_hashes(local),
             "verbose": a.verbose}
-    batcher = Batcher(lambda reqs, pq: m.decide_batch(reqs, pq, usage=True), max_batch=a.max_batch)
+    def run(reqs, pq):
+        res = m.decide_batch(reqs, pq, usage=True)
+        return [(a_, t_, k_) for (a_, t_), k_ in zip(res, m.last_image_tokens, strict=True)]
+
+    batcher = Batcher(run, max_batch=a.max_batch)
     key = os.environ.get("QEV_API_KEY") or None
     httpd = Server((a.host, a.port), make_handler(batcher, info, a.encoding, key))
     print(f"qev_server: {info['backend']} backend on {info['device']}, ready in {info['load_s']} s; "