LightOnOCR-2-1B-DSpark

A DSpark speculative-decoding drafter for lightonai/LightOnOCR-2-1B. It makes page OCR ~2.6× faster end to end under greedy decoding in vLLM.

• Architecture: DSpark, which is the DFlash block drafter plus a Markov head (rank 256) and a confidence head. It has 3 Qwen3 layers (2 sliding-window, 1 full attention), block size 8, the full 151,936-token vocabulary, and 245 MB of bf16 weights.
• Target features: verifier layers 2, 14 and 25, plus the last layer.
• Training: offline with speculators 0.9.0, on hidden states of LightOnOCR-2-1B's own greedy outputs for 19.6K pages from olmOCR-mix-1025. It ran for 4 epochs on one RTX PRO 6000.

Usage (vLLM ≥ 0.33)

vllm serve lightonai/LightOnOCR-2-1B \
  --limit-mm-per-prompt '{"image": 1}' --mm-processor-cache-gb 0 --no-enable-prefix-caching \
  --speculative-config '{"model": "nyamkhuu/LightOnOCR-2-1B-DSpark", "num_speculative_tokens": 7}'

Send requests exactly as the base model card shows: an image-only user message, with pages rendered at 1540 px on the longest side.

Results

All numbers come from 300 held-out olmOCR-bench pages, stratified across its 7 categories (56 pages at concurrency 1). Decoding is greedy (temperature 0) with natural output lengths, and each run is compared with no-spec decoding on the same pages. The setup was vLLM v0.33.0 on one RTX PRO 6000 Blackwell with max_model_len 8192. τ is the mean number of tokens committed per target step, so a value of 1 means no speedup.

K (draft tokens)	Concurrency	τ	Page latency speedup	Throughput speedup	TPOT ms (no-spec → spec)
7	1	5.59	2.66×	2.66×	2.37 → 0.75
7	16	5.40	2.59×	2.44×	6.58 → 2.17
7	32	5.46	2.59×	2.51×	11.15 → 3.75
4	1	4.35	2.22×	2.22×	2.37 → 0.95
4	16	4.25	2.28×	2.22×	6.58 → 2.61
4	32	4.27	2.33×	2.29×	11.15 → 4.40

K=7 is faster than K=4 at every concurrency tested.

With the base card's sampling settings (temperature 0.2, top_p 0.9, with no-spec also sampled), on the same pages and cells:

K	Concurrency	τ	Page latency speedup	Throughput speedup	TPOT ms (no-spec → spec)
7	1	5.20	2.40×	2.40×	2.32 → 0.76
7	16	5.38	2.45×	2.45×	6.74 → 2.32
7	32	5.40	2.56×	2.43×	11.38 → 3.93
4	1	4.28	2.11×	2.10×	2.32 → 0.96
4	16	4.24	2.20×	2.18×	6.74 → 2.76
4	32	4.23	2.33×	2.24×	11.38 → 4.56

At this low temperature sampling costs little. Against greedy, τ drops by 0.5–1% at concurrency 16 and 32 and by 7% at concurrency 1. Page speedup drops by 1–5% at 16 and 32 and by 10% at concurrency 1, which is the smallest and noisiest cell (56 pages).

Throughput. This is greedy steady-state output tokens/s, counted while every request slot was busy. The c=64 and c=128 runs used 600 requests and max-num-seqs 128.

Concurrency	no-spec	K=4	K=7
16	2,378	5,436	6,199
32	2,852	6,746	7,558
64	3,021	7,034	7,890
128	3,185	6,006	6,908

No-spec throughput is nearly flat past concurrency 32, because image prefill dominates. With K=7, peak throughput is 2.5× no-spec's peak (7.9K vs 3.2K tokens/s at concurrency 64 vs 128). Spec throughput falls from c=64 to c=128 but is still 2.2× no-spec there.

τ by olmOCR-bench category at concurrency 16, K=7:

arxiv_math	headers_footers	long_tiny_text	multi_column	old_scans	old_scans_math	tables
5.84	4.92	4.92	5.88	4.12	5.46	6.67

Offline validation on a random 2.5% of the training conversations (teacher-forced, not held out by document) gave an acceptance length of 6.69 and accuracy of 0.97 at draft position 1 and 0.69 at draft position 8 (val_metrics.json).

Output agreement

Greedy speculative decoding matches no-spec output exactly on 80–85% of pages. This rate is the same as no-spec compared against itself at a different concurrency (79–87%). bf16 batch-shape nondeterminism flips near-tie tokens in long OCR outputs, so the drafter adds no measurable disagreement of its own beyond that. When pages do differ, the median character similarity is 0.999.

Limits

• Throughput was measured up to concurrency 128. Spec throughput peaks near c=64; the cause of the drop at c=128 (KV-cache pressure or verification compute) was not isolated.
• Sampling was measured only at the base card's settings (temperature 0.2, top_p 0.9). Higher temperatures will lower acceptance.
• Only one GPU type was tested.

Training data

The training data is LightOnOCR-2-1B's own greedy transcriptions (on-policy) of 20.5K single-page PDFs. They were sampled from four subsets of allenai/olmOCR-mix-1025 (ODC-BY): documents, books, Library of Congress transcripts, and National Archives. Pages were rendered at 1540 px on the longest side. Train and validation were split at random (97.5% / 2.5% of conversations), not by source document. PDFs byte-identical to any olmOCR-bench PDF were removed. 4.3% of outputs hit the 4,096-token limit and were dropped. The loss is {"ce": 0.1, "tv": 0.9}, with lr 6e-4 on a cosine schedule and 32,768-token batches. The full command is in training_command.txt.

License

Apache-2.0, the same as the base model. The training data is derived from olmOCR-mix-1025 (ODC-BY, Allen Institute for AI).