Nitrofurantoin is not a disease: OpenMed's three open models for finding diseases in English medical text

Run a disease extractor over a discharge summary and read what comes back.

Our test note is synthetic: a 72-year-old woman admitted with heart failure, a urinary tract infection treated with nitrofurantoin, and breast cancer years ago, treated with lumpectomy and radiotherapy. At a threshold of 0.5, the original GLiNER2.5 base model returned "nitrofurantoin" as a disease, with a score of 0.63. It also returned "lumpectomy" and "radiotherapy". An antibiotic and two treatments, each with a confidence score that looks like any other.

A wrong disease tag is easy to forgive in a demo. In a pipeline it lands in a problem list, a cohort filter or a coding suggestion, and someone downstream has to catch it.

Our fine-tune of the same model, at the same threshold, dropped all three. It also kept "osteoarthritis of both knees" whole, where the original cut it to "osteoarthritis". Exact spans matter when the next step is matching text to a code.

One note proves nothing on its own. This article covers the whole family we released in October: three open models that find disease mentions in English biomedical and clinical text and return exact character offsets with a confidence score. How we tested them, what improved, where each one fails, and which one to pick.

The family

All three have one label (disease), work in English, are released under Apache-2.0, and run on CPU through the OpenMed SDK.

How we tested them

We trained on one corpus and tested on another.

• Training: the official NCBI Disease training split, 592 unique PubMed abstracts after removing one duplicate. Each model's threshold was chosen on NCBI's official 100-abstract development split.
• Test: the BC5CDR test set, 500 PubMed abstracts with 4,424 disease mentions labeled by people. A different team annotated it, with different guidelines. We checked it against every NCBI split by document ID, normalized text and near-duplicate shingles, and found no overlap.
• Scoring: exact character spans. A prediction with its edges off counts twice: once as a wrong tag and once as a miss.
• Freezing: checkpoints, thresholds and evaluation code were locked before the test set was opened. One seed (42), four epochs, batch size 8. No test-driven retries.
• Uncertainty: a paired document bootstrap with 2,000 draws. These intervals cover document sampling, not training-seed variation.

GLiFormer Base came first, in a separate round. Its before-and-after comparison was run on NCBI Disease's own held-out test split (100 abstracts, 960 mentions). In the second round we scored it on BC5CDR as a reference, but we did not run its original Knowledgator checkpoint there.

Fewer wrong tags, fewer misses

On BC5CDR at threshold 0.5, the fine-tuned GLiNER2.5 Base made 1,227 wrong disease tags, down from 2,245, and missed 1,575 mentions, down from 2,039. GLiFormer Large went from 1,275 wrong tags to 1,029, and from 2,265 misses to 1,551.

All scores below use threshold 0.5 for both the original and the fine-tune.

Precision and recall rose for every model. GLiNER2.5 Base gained more in precision, which fits the note above: fewer drugs and treatments tagged as diseases. The two GLiFormer models gained more in recall.

Every interval sits above zero, at the fixed threshold and at each model's dev-chosen threshold.

At each model's own default threshold, the BC5CDR F1 is 65.84 for GLiNER2.5 Base, 68.42 for GLiFormer Large and 63.51 for GLiFormer Base.

Expect lower scores outside the training corpus. On NCBI's development split, GLiNER2.5 Base went from 56.11 to 87.52 F1. On BC5CDR it went from 53.31 to 65.84. The gain holds on text written by someone else, and it is smaller there.

Same note, three models

Back to the synthetic discharge summary. Through openmed.analyze_text on CPU, at their default thresholds, the three models return 26, 26 and 24 mentions in 0.22, 0.30 and 0.44 seconds.

Not every mention is a disease in the strict sense. At threshold 0.5, the original GLiFormer Large returned 18 spans from this note and our fine-tune returned 25. The new ones include dyspnea, orthopnea and bilateral leg swelling, which are symptoms; pulmonary edema and a right pleural effusion, which are findings; and acute coronary syndrome and stroke, which the note rules out. For a heart failure cohort, that is the story you want surfaced. For a strict disease registry, it is noise to filter.

Negation is a separate step. With assert_context=True, the SDK labels each mention as negated, about a family member, or in the past. On the same note, GLiFormer Base's 26 mentions come back as 19 current, 4 past, 1 family and 2 negated.

The context step is rule-based and can miss. In "No history of asthma or epilepsy", epilepsy comes back affirmed.

Speed

GLiNER2.5 Base is the fast one. Measured side by side on the same hardware, in one batch pass with model load excluded, it processed about 37 times more abstracts a second than GLiFormer Base or Large, and it needed the least memory: 0.99 GiB at peak, against 1.25 and 2.58 GiB.

That gap is not a fair verdict on GLiFormer. The optional flashdeberta package was not installed, so it fell back to eager attention. These numbers compare the native runtimes we tested, not optimized serving stacks, and each is one measured pass.

On CPU, through the SDK, GLiNER2.5 Base ran the 12 synthetic example notes from its card in 0.76 seconds, 55 to 70 ms each.

Two things that went wrong first

A default that overrode the protocol. The first GLiFormer Base training run showed far more steps than four epochs should take. An inherited max_steps=10000 default had overridden num_train_epochs=4. We canceled it before any test evaluation and kept its job in the ledger. The corrected run restarted from the same pinned checkpoint with max_steps=-1, checked the resolved four-epoch configuration before the first step, and checked again that it stopped at epoch four. It took 904 steps.

A pilot that disagreed with itself. Our ten-step pilot trains briefly, saves, reloads in a fresh process and checks that the confidence scores match. In the second round it failed. The native trainer had switched PyTorch's global TF32 precision setting, so inference after training ran at a different precision than in a fresh process. Forcing highest-precision inference fixed it, and the corrected pilot reloaded with zero score difference. We discarded the pilots and trained from the pinned upstream checkpoints.

There was a smaller catch in the data. 68 training windows cut a labeled disease in half, and 44 more could not keep exact gold boundaries with GLiNER2.5's word splitter. We removed them from training only, leaving 1,758 windows for both second-round models. The test labels were never filtered.

Blind spots, model by model

GLiNER2.5 Base: bare abbreviations. On "PMH: COPD, CHF, T2DM, OSA on CPAP. Admitted for an acute exacerbation of COPD." it returns only "acute exacerbation of COPD". GLiFormer Base finds all four abbreviations. It also cuts "diffuse large B-cell lymphoma" down to "B-cell lymphoma", and misses "Latent tuberculosis" at its default threshold.

GLiFormer Base: the weakest of the three on BC5CDR, at 64.10 F1. It returns some symptoms and findings, and can split a long phrase into "heart failure" and "reduced ejection fraction".

GLiFormer Large: header noise and the odd miss. It tags the section header "PMH" as a disease, with a score of 0.86. On its card examples it misses "drug-induced liver injury", and "heart failure" in the first line of the long note.

One thing all three get right: on "Started metformin 500 mg twice daily and lisinopril 10 mg once daily", they return nothing. At threshold 0.5, the original GLiFormer Base returned metformin (0.74) and lisinopril (0.58) as diseases. Our fine-tune, at the same threshold, returns nothing.

Which one to pick

• Every note in a hospital, at volume: GLiNER2.5 Base. Watch for problem lists written only in abbreviations.
• Abbreviation-heavy problem lists: GLiFormer Base.
• The best accuracy, where throughput allows: GLiFormer Large. At each model's default threshold, it leads GLiNER2.5 Base by 2.58 F1 points on BC5CDR, with a descriptive paired 95% interval of +1.22 to +3.97 that is not adjusted for multiple comparisons. Picking a winner on this test is a review decision, not a second independent validation.

What it is not

• Trained and tested on PubMed abstracts. Not validated on hospital notes. Every note in this article is synthetic.
• One training seed. The intervals above do not capture seed-to-seed variation.
• Whether the upstream base models saw these public benchmarks in their own training is unknown.
• No normalization: spans are not mapped to ICD-10, SNOMED CT or MeSH codes.
• English only, one label. The base models' other tasks were not evaluated after fine-tuning.
• They find text spans. They do not diagnose anything, and none of them is a medical device.

Try it

The OpenMed SDK is Apache-2.0 and all three models run on CPU. For GLiNER2.5 BioNER Disease Base:

pip install -U "openmed[gliner2]"  # OpenMed 3.2 or later

from openmed import analyze_text

text = (
    "65-year-old man with type 2 diabetes mellitus, stage 3 "
    "chronic kidney disease and atrial fibrillation, "
    "admitted with community-acquired pneumonia."
)
result = analyze_text(
    text,
    model_id="OpenMed/GLiNER2.5-BioNER-Disease-Base-v1",
    labels=["disease"],
    threshold=0.7,
)
for e in result.entities:
    print(e.start, e.end, e.text, f"{e.confidence:.2f}")

21 45 type 2 diabetes mellitus 0.98
47 77 stage 3 chronic kidney disease 0.82
82 101 atrial fibrillation 0.99
117 145 community-acquired pneumonia 0.90

For the GLiFormer models, install openmed[gliformer] instead and set model_id to OpenMed/GLiFormer-BioNER-Disease-Base-v1 (threshold 0.7) or OpenMed/GLiFormer-BioNER-Disease-Large-v1 (threshold 0.6). Add assert_context=True to get the negation, family and timing labels shown above.

Every example and every known miss is on the model cards: GLiNER2.5 BioNER Disease Base, GLiFormer BioNER Disease Base and GLiFormer BioNER Disease Large.

Thank you

These models stand on open work. Thank you to Fastino for GLiNER2.5 base and the GLiNER2 library, and to Knowledgator for GLiFormer base and large and the GLiFormer library. Thank you to Rezarta Islamaj Doğan, Robert Leaman and Zhiyong Lu for the NCBI Disease corpus, and to Jiao Li, Zhiyong Lu and the BioCreative V CDR team for BC5CDR. Releasing corpora like these is what makes a cross-corpus test possible at all.

These models find text spans. They do not make clinical decisions. They should, though, leave the antibiotics out of the disease list.