abhiprd2000/bhojpuri-sentiment-model

Bhojpuri Sentiment Model (v2)

4-class sentiment classifier for native Bhojpuri text (Devanagari script), fine-tuned from ai4bharat/IndicBERTv2-MLM-only on majority-vote consensus labels from 5 open-weight LLM judges (see the PURVA Bhojpuri Sentiment Corpus).

Labels

objective, positive, negative, neutral — see the dataset card for full definitions. objective reports fact with no evaluative stance; positive/negative/neutral apply to subjective sentences by polarity.

Training

Base model	ai4bharat/IndicBERTv2-MLM-only (revision 8598f13)
Label source	majority vote across 5 LLM judges (not Dawid–Skene — see dataset card for why)
Train / dev / test	72,168 / 9,019 / 9,020 (stratified by register × text_type × source_name, seed 42)
Epochs	4
Learning rate	2e-5
Batch size	32
Max sequence length	128
Precision	fp16
Seed	42
Checkpoint selection	final epoch (no best-on-dev selection; dev loss used for monitoring only)

Results

Scored on a 108-item human-labeled gold set (gold_test_120.jsonl minus 3 mixed-labeled items, which fall outside this model's 4-class label space; verified zero overlap with train/dev). Bootstrap 95% CI, 1,000 resamples.

Metric	Score	95% CI
Accuracy	65.7%	[56.5%, 75.0%]
Macro-F1	50.3%	[41.3%, 59.5%]

By register:

Register	n	Accuracy
encyclopedic	12	0.833
news	42	0.810
literary	19	0.632
verse	18	0.500
commentary	15	0.333
opinion	2	n<10, not reported

Accuracy is strong on news/encyclopedic register and weak on commentary and verse — see the dataset card's "Known issues" for the neutral-class and Dawid-Skene-vs-majority-vote caveats, which apply to this model too (it is the majority-vote-trained checkpoint; a separate Dawid-Skene-trained checkpoint scores 63.0% accuracy / 51.4% macro-F1 on the same gold set — within this comparison's confidence interval, not a clear win either way).

Usage

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="abhiprd2000/bhojpuri-sentiment-model",
)

texts = [
    "ई खाना बहुत स्वादिष्ट बा।",       # positive (99.8%)
    "आज बहुत थकान लागत बा।",            # negative (99.6%)
    "हम कल पटना जाइब।",                 # objective (95.7%)
]

for text in texts:
    result = classifier(text)[0]
    print(f"{text} -> {result['label']} ({result['score']*100:.1f}%)")

Licence

MIT, matching the base model (ai4bharat/IndicBERTv2-MLM-only, confirmed license:mit on its model card). Differs from v1-legacy, which was tagged apache-2.0.

Version history

• v1-legacy (tag v1-legacy): XLM-RoBERTa (cardiffnlp/twitter-xlm-roberta-base-sentiment) fine-tuned on a 68,822-row, artificially class-balanced, partly machine-translated dataset. Scored 97.6% accuracy / 0.976 macro-F1 on its own held-out 501-item balanced/translated test set.
• v2 (this model): IndicBERTv2, trained on native non-translated text with naturally imbalanced labels, scored against a human-labeled gold set. v1 and v2 numbers are not comparable — different base model, different training data, different label source, different test set. Do not quote v1's 97.6% alongside v2's 65.7% as if they measure the same thing.