Quazim0t0/Byrne-Docling-131M

Byrne-Docling (131M)

Byrne-Docling is a document-understanding variant of the Byrne-VLM family: a ~131M from-scratch SpikeWhale / Byrne vision-language model trained to convert document images into DocTags (the structured markup used by Granite-Docling / SmolDocling — <chart>, <formula>, <code>, OTSL <fcel> table cells, <loc_N> bounding boxes, code-language tags like <_Python_>, …).

Research artifact — it produces clean DocTags structure and reads coarse content, but is not a production document parser.

document image → 448px letterbox → Byrne-VE (frozen, 784 tokens)
              → Connector → Byrne LM + Family-LoRA (+ atomic DocTags tokens) → DocTags

Current architecture

part	detail
Byrne-VE	39M ViT-style encoder, 448px / patch16 → 784 tokens, RMSNorm, 2D-axial RoPE, QK-Norm, SwiGLU, HRM. DINOv2-distilled → DINO self-distilled.
Preprocessing	letterbox (aspect-preserving pad) — the whole page is visible (center-crop was cutting ~90% off tall documents).
Connector	2-layer MLP (512→640).
Byrne LM	~90M SpikeWhale LM, custom SpikeTokenizer, 4096 ctx.
Family-LoRA	HRM + MoE-SwiGLU adapter on the LM decoder.
DocTags tokens	140 DocTags markup tokens added to the tokenizer as ATOMIC tokens (vocab 16512→16652). Each tag (<fcel>, <chart>, <_C++_>, <loc_0>…) is now one token the model emits correctly-or-not — instead of spelling it char-by-char. Only these 140 new embedding rows were trained on top of the frozen base LM.

How we got here (the honest development story)

This model went through several iterations; each fixed a specific failure the previous one exposed.

v1 — 224px + AnyRes tiling (char-level DocTags). The first Byrne-Docling used a 224px encoder with AnyRes 2×2 tiling (5 tiles → 980 image tokens) to raise effective resolution, trained on 6,000 ground-truth (image → DocTags) pairs from the SmolDocling synthetic datasets (SynthChartNet, SynthFormulaNet, SynthCodeNet) via connector + Family-LoRA. It learned the DocTags format and coarse content (e.g. chart column headers, C++ copyright, LaTeX), but greedy decoding tended to loop, so a repetition penalty is used at inference. This established the pipeline.

Higher-resolution encoder (v2, 448px). To read finer detail we distilled a native 448px / 784-token encoder (DINOv2 distill → DINO self-distill). Naively swapping it in with center-crop preprocessing made things worse — malformed tags and cross-modal confusion (code emitted inside <formula>). Diagnosis: center-crop was cropping tall document pages down to a square, discarding ~90% of the content.

Letterbox preprocessing. Replacing center-crop with aspect-preserving letterbox padding (whole page visible) fixed the regression: correct content and modality (formula→LaTeX, code→right language) returned. One weakness remained — tag syntax was noisy (<fcel> came out as <fcil> / <fbar>).

Atomic DocTags tokens (this release). The tag noise was a tokenization problem, not training or resolution: DocTags tokens weren't in the vocab, so the model spelled them character-by-character and a single wrong subword corrupted the tag. Adding the 140 DocTags tokens to the tokenizer as atomic tokens (and training just their new embedding rows) cleaned up the syntax — proper <chart><loc_0><loc_0><loc_500><loc_500> <bar_chart><fcel>…<nl></chart>, correct </formula> closings, clean <_Java_> language tags. This is the same trick Granite-Docling itself uses.

Evaluation (honest)

On held-out chart / formula / code, the current model produces clean DocTags structure (proper <fcel>/<chart>/<loc_N>/<nl> + closing tags) with correct modality and coarse-correct content (real financial labels, LaTeX, license headers). Remaining errors — repeated values, occasional wrong specifics — are the capacity limits of a 39M encoder + 90M LM on 6k synthetic pairs, not tokenization. Use a repetition penalty at inference.

Usage

import torch
from generate import load_vlm, caption
from spike_tokenizer import SpikeTokenizer

dev = "cuda" if torch.cuda.is_available() else "cpu"
tok = SpikeTokenizer(vocab_file="tokenizer_doctags.json")
# load_vlm reads new_vocab from the checkpoint and resizes the LM automatically
vlm = load_vlm("weights/byrne_docling.pt", "lm", "weights/byrne_ve.pt", dev)
doctags = caption(vlm, tok, "page.png", dev, max_new=256,
                  repetition_penalty=1.2, no_repeat_ngram=3, letterbox=True)
print(doctags)

CLI:

python generate.py --image page.png --ckpt weights/byrne_docling.pt \
  --vision-ckpt weights/byrne_ve.pt --tokenizer tokenizer_doctags.json \
  --letterbox --max-new 256 --repetition-penalty 1.2 --no-repeat-ngram 3

Files

weights/byrne_docling.pt (connector + Family-LoRA + trained DocTags embeddings) · weights/byrne_ve.pt (448px letterbox encoder) · lm/ (Byrne base LM) · tokenizer_doctags.json (vocab 16652) · model code.

Citation

@misc{byrne_docling_2026,
  title  = {Byrne-Docling: A Tiny SpikeWhale VLM for Document DocTags (letterbox + atomic tokens)},
  author = {Quazim0t0},
  year   = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Quazim0t0/Byrne-Docling-131M}}
}

License

Apache-2.0.

Escarda vs Byrne — vision family comparison

The Byrne family uses HRM refinement. Escarda = Byrne + JEPA (Joint-Embedding Predictive head) added alongside HRM in both the vision encoder and the LM trunk — auxiliary only, zero inference cost.

Vision encoder (DINOv2 teacher-alignment, n=1024 held-out):

	Byrne-VE	Escarda-VE
Params	39.34M	39.60M (+JEPA head)
CLS cosine	0.776	0.771
PATCH cosine	0.600	0.584
JEPA self-consistency	—	0.040

Docling (same held-out doc images, atomic DocTags): both emit well-formed DocTags; Byrne-Docling is marginally more complete on the hardest samples (closes </formula>, includes the <code> wrapper), consistent with its slightly higher teacher-alignment. Escarda-Docling is structurally on par and adds the JEPA representation-learning trait.

Pros/cons. Byrne (HRM): higher teacher-alignment, all capacity on distillation fidelity; no self-supervised objective. Escarda (HRM+JEPA): self-supervised neighbour-prediction (richer spatial structure) at zero inference cost, trading ~1–3% teacher-alignment. Same size class.

Family repos: Byrne-VE · Escarda-VE · Byrne-Docling-131M · Escarda-Docling-126M

Engram repair (update)

The n-gram Engram memory in this checkpoint was degenerate: every token hashed to bucket 0, so only 1 of 4096 rows ever trained and the memory was a single constant (no further training could fix it). This version applies a behavior-preserving repair (rescale the frozen hash compressor + broadcast bucket 0) — outputs are unchanged — then a short distill with the Engram tables unfrozen so the now-reachable buckets populate.

Verified: hash bucket usage went from 1/4096 → 4096/4096, and the Engram tables are now a genuinely populated n-gram memory instead of a constant. DocTags output remains valid.