decosa-number-checker-modernbert-large (v2)

Given a claim sentence and its evidence (a table, a passage or both), it says whether the claim's numbers are supported, contradicted or not verifiable. For a contradiction it names the error type (wrong value, unit or scale, period, direction, arithmetic, or entity) and points at the evidence cell it read.

It is the learned tier of a deterministic number check. The code in numparse.py extracts every figure in the claim (amounts with scale words, percentages, percentage points, fractions such as "about a fifth", multiples such as "nearly doubled") and searches the evidence for a derivation: a cell, a change, a percentage change, a share, a sum, a point change. The model reads the claim, those findings and the evidence, and decides what the code cannot: whether the derivation is about the right line, period and direction, and what an untraced figure means.

• Weights: Apache-2.0. Base: ModernBERT-large (Apache-2.0).
• Size: 395M parameters; model.safetensors 1.58 GB; runs on a CPU (median 453 ms a claim on 4 threads; v1 on ModernBERT-base took 135 ms).
• sha256 of every file: SHA256SUMS; the evaluation numbers below are also in eval_summary.json.
• Versions: v2 is the first public release. v1 was an earlier internal model on ModernBERT-base; it is shown in the tables for reference and is not released.
• It needs this repository's modeling_checker.py and numparse.py (custom heads and the code tier); it is not a stock transformers pipeline.

Use

# pip install torch transformers safetensors
from modeling_checker import NumberChecker   # this repository's modeling_checker.py and numparse.py

nc = NumberChecker.from_dir(".")
nc.check("Revenue rose by about a fifth in 2020.",
         table=[["", "2019", "2020"], ["Revenue", "1,000", "1,210"]], unit="millions")
# {'verdict': 'supported', 'p': {...}, 'code': 'traced', 'error_type': None, 'pointer': {...}}

nc.check("Revenue fell 21% in 2020.", table=[["", "2019", "2020"], ["Revenue", "1,000", "1,210"]])
# {'verdict': 'contradicted', 'code': 'mismatch', 'error_type': 'direction', ...}

Rules we use in production, and recommend:

• A hard code mismatch is final (code == "mismatch": a thousand-fold slip against a declared unit, the other period's figure for the same line, a signed change stated in the wrong direction). The model never overrules it.
• When the code traced every figure, flag the claim only when the model is very sure (0.813 or more on "contradicted").
• Otherwise take the model's answer at 0.78 or more, and send the rest to a person or a larger model.

onnx/model.onnx is the same network exported for ONNX Runtime (inputs input_ids, attention_mask; outputs verdict, error, pointer logits). Divide the verdict logits by temperature in checker.json before the softmax.

Training data

About 80,000 claim/evidence pairs, built by us. No non-commercial data was used at any stage, and nothing from QuanTemp (CC BY-NC).

Source	Licence	What we used
SEC Financial Statement Data Sets 2026q1 (EDGAR XBRL face statements, 4,489 filings)	US government work, public domain (17 U.S.C. 105)	statement tables; claims written and planted by our generator
FinQA (github.com/czyssrs/FinQA)	MIT	tables and passages from its train split; claims by our generator
TAT-QA (github.com/NExTplusplus/TAT-QA)	CC BY 4.0 (data), MIT (code)	tables and passages from its train split; claims by our generator
TabFact (github.com/wenhuchen/Table-Fact-Checking)	MIT (dataset); tables from Wikipedia, CC BY-SA	10,000 human claims with a digit, train split
ClinicalTrials.gov results (API v2, 439 phase 3 interventional trials with results, retrieved 27 Sep 2026)	US government, public domain; attribution and a statement of changes below	results and adverse-event tables; lay-language claims by our generator
Decosa synthetic SAR case files	ours (generated from a seed by code; fictional people, accounts and banks)	ledger rows and narrative sentences, with the numeric block's own findings

FinQA's tables come from FinTabNet (IBM, CDLA-Permissive-1.0), which is built from S&P 500 companies' annual reports; TAT-QA's tables and passages also come from public financial reports; TabFact's claims were written by crowd workers. No training text was written by a closed-API model; none of the training data ships with this repository.

ClinicalTrials.gov. Data retrieved from ClinicalTrials.gov (National Library of Medicine), API v2, on 27 September 2026. Changes: we parsed the results and adverse-event tables, cut them to the rows a claim concerns, and generated claim sentences about them, some with planted numeric errors. The results were not reviewed by NLM, and ClinicalTrials.gov does not endorse this model.

Contradictions were planted in code with known labels: a value changed by 3-30%, a thousand-fold scale slip, the other period's figure, a flipped rise/fall, rounding beyond the stated precision, percent written for percentage points, a percentage on the wrong base, a sum missing a term, another line's or group's figure. Not-verifiable pairs remove the claim's rows from the evidence. TabFact adds human-written claims (entailed or refuted) over Wikipedia tables.

Evaluation

Held out: other filings, other trials, other SAR seeds and the official test splits; never trained or tuned on. Thresholds were set on the dev split. TabFact's claims are human-written and the TAT-QA answers are real question-answer pairs over real report tables, so those two sets are the closest to real use; the other three use planted errors from our generator. "Caught" is recall on contradicted claims; "false flags" is the share of supported claims called contradicted. The LLM judge is Qwen3.8-27B (temperature 0), on a random sample of each set.

Set	System	n	Accuracy	Caught	False flags
Planted errors, financial tables (EDGAR, FinQA, TAT-QA test splits)	code only	3000	59.8%	20.9%	0.1%
Planted errors, financial tables (EDGAR, FinQA, TAT-QA test splits)	this model	3000	97.9%	97.8%	2.6%
Planted errors, financial tables (EDGAR, FinQA, TAT-QA test splits)	code, then this model	3000	94.3%	96.2%	1.7%
Planted errors, financial tables (EDGAR, FinQA, TAT-QA test splits)	Qwen3.8-27B judge (random sample)	400	78.2%	85.3%	27.4%
Planted errors, financial tables (EDGAR, FinQA, TAT-QA test splits)	v1, internal ModernBERT-base (not released), for reference	3000	97.5%	96.7%	2.3%
Planted errors, clinical-trial results (held-out CT.gov trials)	code only	520	51.9%	0.0%	0.0%
Planted errors, clinical-trial results (held-out CT.gov trials)	this model	520	96.9%	93.3%	0.4%
Planted errors, clinical-trial results (held-out CT.gov trials)	code, then this model	520	95.6%	92.9%	0.4%
Planted errors, clinical-trial results (held-out CT.gov trials)	Qwen3.8-27B judge (random sample)	178	86.5%	77.9%	8.9%
Planted errors, clinical-trial results (held-out CT.gov trials)	v1, internal ModernBERT-base (not released), for reference	520	90.4%	91.1%	11.3%
Planted errors, synthetic bank ledgers (held-out seeds)	code only	2727	96.0%	98.9%	0.0%
Planted errors, synthetic bank ledgers (held-out seeds)	this model	2727	99.6%	98.9%	0.0%
Planted errors, synthetic bank ledgers (held-out seeds)	code, then this model	2727	96.0%	98.9%	0.0%
Planted errors, synthetic bank ledgers (held-out seeds)	Qwen3.8-27B judge (random sample)	200	78.5%	55.3%	5.4%
Planted errors, synthetic bank ledgers (held-out seeds)	v1, internal ModernBERT-base (not released), for reference	2727	99.6%	98.9%	0.0%
TabFact test, human claims with a digit	code only	1999	22.6%	0.1%	0.0%
TabFact test, human claims with a digit	this model	1999	73.8%	68.1%	20.8%
TabFact test, human claims with a digit	code, then this model	1999	72.6%	60.7%	16.2%
TabFact test, human claims with a digit	Qwen3.8-27B judge (random sample)	86	83.7%	81.6%	12.5%
TabFact test, human claims with a digit	v1, internal ModernBERT-base (not released), for reference	1999	65.5%	61.6%	30.8%
TAT-QA test gold answers restated as sentences by Qwen3.8-27B	code only	800	32.3%	4.3%	0.5%
TAT-QA test gold answers restated as sentences by Qwen3.8-27B	this model	800	79.4%	87.3%	28.2%
TAT-QA test gold answers restated as sentences by Qwen3.8-27B	code, then this model	800	79.7%	86.3%	26.5%
TAT-QA test gold answers restated as sentences by Qwen3.8-27B	Qwen3.8-27B judge (random sample)	300	60.0%	40.4%	18.2%
TAT-QA test gold answers restated as sentences by Qwen3.8-27B	v1, internal ModernBERT-base (not released), for reference	800	41.4%	47.0%	21.0%

v2 (ModernBERT-large, 2 epochs, 512 tokens) is more accurate than v1 on four held-out sets and level on the synthetic ledgers; it has more false flags on the TAT-QA set (28.2% vs 21.0%) and is about 3.4 times slower on a CPU. On free-form claims it closed most of the gap: TAT-QA answers restated in plain words 79.4% (v1 41.4%; Qwen3.8-27B 60.0% on a 300-claim sample where this model is at 79.0%; Qwen3.8-27B also wrote these sentences), TabFact human claims 73.8% (v1 65.5%; Qwen3.8-27B 83.7% on the same 86-claim sample where this model is at 67.4%). Counting and superlatives remain the LLM's ground. Treat it as a cheap first reader for number-bearing sentences that cite their table, not as a general fact checker.

Limitations

• Trained on planted errors. Its strength is the error kinds our generator plants; accuracy on real drafting errors is not measured. Planted financial sets carry some label noise from table quirks (undeclared units, oddly formatted cells).
• Free-form claims are weaker: TabFact human claims 73.8% with 20.8% false flags; TAT-QA plain-word answers 79.4% with 28.2% false flags (higher than v1's 21.0%).
• A learned bias we found: in our clinical-trial training data, "no one in group X died" appeared only as a planted error, so the model tends to call such sentences contradicted. Check "no one" and "none" claims with code, not with this model.
• Reads English only; tables of up to about 10 rows are shown to it (longer tables are cut to the rows the code and the claim touch).
• It checks claims against the evidence given, not against the world. It does not handle dates as well as amounts and percentages.
• It flags and ranks; it should not clear anything on its own in a regulated workflow.

Intended use

A pre-filter and second reader for number-bearing sentences in drafts that cite their tables: financial narratives (MD&A, reports), clinical-trial lay summaries, investigation narratives. It flags and ranks; a person decides. It is not a compliance tool, it does not check claims against the world (only against the evidence given), and it should not clear anything on its own in a regulated workflow. For clinical text it checks that a sentence matches the trial's reported tables; it is not medical advice and must not be used to make medical or treatment decisions.

Model

ModernBERT-large (answerdotai/ModernBERT-large, Apache-2.0) with three heads on the first token and the tokens: verdict (3 classes), error type (6), evidence pointer (per token). 512-token input: the claim, then the code tier's findings, then the evidence with cell ids. 2 epochs on 81,032 pairs, lr 3e-5, batch 32, on one rented A100 (about 36 minutes). Dev split: accuracy 91.8%, error type 97.6%, pointer 96.6%. Temperature 1.25 and the two thresholds were set on dev. ONNX export parity: max probability difference 5e-5, 20 of 20 argmax agree.

Files: model.safetensors (fp32 encoder plus heads), config.json (the encoder), tokenizer files, checker.json (labels, temperature, thresholds), modeling_checker.py, numparse.py, onnx/model.onnx, eval_summary.json, LICENSE, NOTICE, SHA256SUMS.

Licence

Apache-2.0 for the weights and code in this repository. The base model is Apache-2.0 (ModernBERT, Answer.AI and LightOn). The training data sources above ask for attribution; it is given here and in NOTICE.

Citation and contact

Decosa, "decosa-number-checker-modernbert-large (v2)", 2026, decosa.ai. Built with planted errors from Decosa's own pipelines; the generator and the held-out sets are not released.