bge-roleplay-judge-onnx

Browser-ready ONNX sentence-quality classifiers for AIGC Chinese text, fine-tuned from BAAI/bge-base-zh-v1.5.

Each sentence is scored in isolation into six classes:

#	标签	含义
1	烂完了	八股味重、语义崩坏，单看这句就要重写
2	小改	有明显问题，修一修还能用
3	平庸	能读，但平庸无趣，缺少打磨
4	还可以	读起来挺顺，可以保留
5	优秀	文笔出色，值得保留
6	跳过	无需判断（章节号、标记、结构化信息、非正文内容）

Class 6 is a separate "skip" head, not a point on the 1–5 quality scale. expected_quality is computed over classes 1–5 after renormalising.

Files

Path	Model	Size	Precision	Spearman	MAE
onnx/8l-int8/model.onnx	8L KD distilled	70.7 MiB	INT8	0.6291	0.7481
onnx/12l-fp32/model.onnx	12L interpolated	388.1 MiB	FP32	0.6406	—

Both are browser-native: the graph uses only ops that onnxruntime-web's WASM backend executes efficiently. The INT8 graph uses DynamicQuantizeLinear + MatMulInteger, so it stays fast without a GPU.

Use 8l-int8 by default. It is 5.5× smaller and only 0.012 Spearman behind, which matters far more for a web download than the accuracy delta.

Intended use

Score and triage Chinese prose sentence by sentence — reviewing long-form AIGC output, finding sentences a human should rewrite, and filtering non-content.

Not suitable for: attributing text to a specific author or model, or as proof that text is machine-generated. The classifier was trained on human quality votes; it scores quality, and correlated style is not evidence of provenance.

Usage

The model takes input_ids and attention_mask (int64) and returns a [batch, 6] logit tensor. Apply the fitted temperature before softmax — it is required for calibrated probabilities.

import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

REPO = "GoForceX/bge-roleplay-judge-onnx"
TEMPERATURE = 2.3511865550320645  # 8l-int8; 2.337502972054673 for 12l-fp32
LABELS = ["烂完了", "小改", "平庸", "还可以", "优秀", "跳过"]

ort_session = ort.InferenceSession(
    "onnx/8l-int8/model.onnx", providers=["CPUExecutionProvider"]
)
tokenizer = AutoTokenizer.from_pretrained("BAAI/bge-base-zh-v1.5")

sentence = "「上课？上什么课？我连他们讲话都听不懂，上哪门子课咯？」"
enc = tokenizer(sentence, truncation=True, max_length=512, return_tensors="np")
logits = ort_session.run(
    ["logits"],
    {
        "input_ids": enc["input_ids"].astype(np.int64),
        "attention_mask": enc["attention_mask"].astype(np.int64),
    },
)[0][0]

z = logits - logits.max()
probs = np.exp(z / TEMPERATURE) / np.exp(z / TEMPERATURE).sum()

quality = probs[:5] / probs[:5].sum()
expected = float((quality * np.arange(1, 6)).sum())
print(LABELS[int(probs.argmax())], round(expected, 3))

Batch composition affects INT8 output

The INT8 graph uses DynamicQuantizeLinear, whose activation scale is derived from the whole input tensor. Padding therefore perturbs results slightly (measured over 240 sentences: probabilities differ by 0.007 on average, expected quality by 0.045). Score one sentence at a time when you need to reproduce the published metrics; batch only for throughput.

Training

Both models start from BAAI/bge-base-zh-v1.5 and share the same head:

last_hidden_state[:, 0] (CLS)
  -> LayerNorm + Dropout(0.1)
  -> Linear(768, 6)
  -> softmax over 6 classes

Objective: human 6-way soft-vote cross-entropy + 0.05 × ordinal loss over classes 1–5. One six-way distribution is used rather than a separate binary skip head, which won an internal ablation. Labels come from 8-vote-per-sentence human annotation.

• 8l (kd-a025-8l-refine767): 6-layer prune of the interpolated 12-layer teacher (layer map [0, 2, 4, 6, 8, 11]), then 8-layer knowledge-distilled refinement (KD weight 0.8, hidden 0.04, low-LR tail update).
• 12l (interp-140-697-a025-v2): weight-space interpolation of an early and a late fine-tuning checkpoint, α = 0.25.

Evaluation is on a 595-sentence held-out split, ranked against human mean quality (Spearman) with expected-quality MAE.

Manifest

manifest.json lists both models with size, SHA-256, fitted temperature and metrics; tokenizer.json is mirrored here for reference. The companion web app reads the manifest and streams weights from this repo, so no .onnx is bundled with the site.