cstr/got-ocr2-crispembed-GGUF

GGUF conversion of stepfun-ai/GOT-OCR2_0 for use with CrispEmbed.

Architecture

• Vision: SAM ViT-B (12 layers, 768d, 12 heads, 16×16 patches, 1024×1024 input)
  • Windowed attention (ws=14) with global attention at layers [2, 5, 8, 11]
  • Decomposed relative position encoding
  • Neck: Conv(768→256) → LN2d → Conv(256→256) → LN2d
  • Downsample: Conv(256→512→1024, stride 2) → 256 vision tokens
  • Projector: Linear(1024, 1024)
• LLM: Qwen2-0.5B (24 layers, 1024d, MHA 16/16, SiLU SwiGLU, RoPE θ=1M)
• Tokenizer: tiktoken (151860 vocab)
• Total: ~0.7B parameters

Files

File	Precision	Size	Notes
got-ocr2-q4_k.gguf	Q4_K	445 MB	Recommended / default. Correct OCR, fastest decode on Apple Silicon
got-ocr2-q8_0.gguf	Q8_0	599 MB	Correct OCR; on M1 the Q8_0 mul_mv path is slower per-token than Q4_K, so Q4_K is preferred
got-ocr2-f16.gguf	F16	1.44 GB	Full precision baseline

Precision & parity

The Qwen2-0.5B decoder quantizes cleanly to Q4_K and Q8_0 — all three builds above produce identical, correct OCR. Verified against the real HF model (transformers GotOcr2) plus a Python f32 reference:

• Vision (ViT layers, neck, downsample, projector): cos ≥ 0.998 vs HF.
• LLM decoder (per-layer, Q8_0 weights vs f32 reference): cos ≥ 0.99996.

Per-token decode speed on an M1 (256 vision tokens spliced into the prompt):

Build	Decode
Q4_K	~20 ms/tok
F16	~38 ms/tok
Q8_0	~42 ms/tok

Q4_K is ~2× faster to decode than F16 and 3× smaller, so it is the default.

Note on earlier builds. A prior version of this repo shipped an F16-decoder build and claimed the 0.5B decoder was "catastrophically sensitive to quantization" (llm_layer_0 cos ≈ 0.936 at Q8_0). That number was a measurement artifact of a per-row bug in the diff harness (it used the token count as the row length), not real quant sensitivity. With the corrected harness the Q8_0/Q4_K decoder matches f32 at cos ≥ 0.99996 and OCR output is identical to F16. See CrispEmbed issue #25.

Usage

crispembed --ocr got-ocr2 image.png

Reproducing the quants

crispembed-quantize got-ocr2-f16.gguf got-ocr2-q4_k.gguf q4_k
crispembed-quantize got-ocr2-f16.gguf got-ocr2-q8_0.gguf q8_0

(The quantizer also has an optional --decoder-f16 flag that keeps the decoder weights at F16; it is not needed for correctness and is retained only for diagnostic / comparison use.)

License

Apache-2.0 (same as upstream model)