adambarbato/PaddleOCR-VL-For-Manga-GGUF

PaddleOCR-VL-For-Manga GGUF

This repo contains BF16 quantized GGUF files of jzhang533's PaddleOCR-VL-For-Manga model.

It can be run via llama.cpp with the following parameters:

llama-server \
  -m ./PaddleOCR-VL-For-Manga-GGUF/PaddleOCR-VL-For-Manga-BF16.gguf
  --mmproj ./PaddleOCR-VL-For-Manga-GGUF/PaddleOCR-VL-For-Manga-mmproj-BF16.gguf
  --host 0.0.0.0
  --n-gpu-layers 999
  --port 9999
  -c 32768
  --temp 0

Send an image along with the prompt OCR: to receive an OCR response. If you enounter this error: tokenize: error: number of bitmaps (1) does not match number of markers (0). send <__media__>OCR: as your prompt and see if that resolves the issue.

Original discussion for PaddleOCR-VL support in llama.cpp can be found here.

Original model card for PaddleOCR-VL-For-Manga is copied below:

PaddleOCR-VL-For-Manga

Model Description

PaddleOCR-VL-For-Manga is an OCR model enhanced for Japanese manga text recognition. It is fine-tuned from PaddleOCR-VL and achieves much higher accuracy on manga speech bubbles and stylized fonts.

This model was fine-tuned on a combination of the Manga109-s dataset and 1.5 million synthetic data samples. It showcases the potential of Supervised Fine-Tuning (SFT) to create highly accurate, domain-specific VLMs for OCR tasks from a powerful, general-purpose base like PaddleOCR-VL, which supports 109 languages.

This project serves as a practical guide for developers looking to build their own custom OCR solutions. You can find the training code at the Github Repository, a step by step tutorial is avaiable here.

Performance

The model achieves a 70% full-sentence accuracy on a test set of Manga109-s crops (representing a 10% split of the dataset). For comparison, the original PaddleOCR-VL on the same test dataset achieves 27% full sentence accuracy.

Common errors involve discrepancies between visually similar characters that are often used interchangeably, such as:

• ！？ vs. !? (Full-width vs. half-width punctuation)
• ＯＫ vs. ok (Full-width vs. half-width letters)
• １２０５ vs. 1205 (Full-width vs. half-width numbers)
• “人” (U+4EBA) vs. “⼈” (U+2F08) (Standard CJK Unified Ideograph vs. CJK Radical)

The prevalence of these character types highlights a limitation of standard metrics like Character Error Rate (CER). These metrics may not fully capture the model's practical accuracy, as they penalize semantically equivalent variations that are common in stylized text.

Examples

#	Image	Prediction
1		心拍呼吸正常値 お人よし度過剰値... 間違いなく パパッ...! 生存確認っ...!
2		あとは『メルニィ 宇宙鉄道』とか 『TipTap』とか 全部その人が 考えたらしい
3		★コミックス20巻1月4日(土)発売〟TVアニメ1月11日(土)放送開始!!
4		我々魔女協会が 長年追い続ける 最大の敵 ウロロが「王の魔法」なら あれは世界を削り変える 「神の魔法」
5		天弓の動きについてくだけじゃ勝てねぇ…！

How to Use

You can use this model with the transformers, PaddleOCR, or any library that supports PaddleOCR-VL to perform OCR on manga images. The model architecture and weights layout are identical to the base model.

If your application involves documents with structured layouts, you can use your fine-tuned OCR model in conjunction with PP-DocLayoutV2 for layout analysis. However, for manga, the reading order and layout are quite different.

Training Details

• Base Model: PaddleOCR-VL
• Dataset:
  • Manga109-s: 0.1 million randomly sampled text-region crops (not full pages) were used for training (90% split); the remaining 10% crops were used for testing.
  • Synthetic Data: 1.5 million generated samples.
• Training Frameworks:
  • transformers and trl
• Alternatives for SFT:
  • ERNIEKit
  • ms-swift

Acknowledgements

• Manga109-s dataset, which provided the manga text-region crops used for training and evaluation.
• PaddleOCR-VL, the base Visual Language Model on which this model is fine-tuned.
• manga-ocr, used in this project for data processing and synthetic data generation; it also inspired practical workflows and evaluation considerations for manga OCR.

License

This model is licensed under the Apache 2.0 license.