metadata
pipeline_tag: translation
language: multilingual
library_name: transformers
base_model:
- FacebookAI/xlm-roberta-large
license: apache-2.0
📊 Estimating Machine Translation Difficulty
This repository contains the SENTINELSRC metric model used for Difficulty Sampling at the WMT25 General Machine Translation Shared Task, and analyzed in our paper Estimating Machine Translation Difficulty.
Usage
To run this model, install the following git repository:
pip install git+https://github.com/prosho-97/guardians-mt-eval
After that, you can use this model within Python in the following way:
from sentinel_metric import download_model, load_from_checkpoint
model_path = download_model("Prosho/sentinel-src-25")
model = load_from_checkpoint(model_path)
data = [
{"src": "Please sign the form."},
{"src": "He spilled the beans, then backpedaled—talk about mixed signals!"}
]
output = model.predict(data, batch_size=8, gpus=1)
Output:
# Segment scores
>>> output.scores
[0.5604351758956909, -0.08413456380367279]
# System score
>>> output.system_score
0.23815030604600906
Where the higher the output score, the easier it is to translate the input source text.
Cite this work
This work has been presented at EMNLP 2025. If you use any part, please consider citing our paper as follows:
@inproceedings{proietti-etal-2025-estimating,
title = "Estimating Machine Translation Difficulty",
author = "Proietti, Lorenzo and
Perrella, Stefano and
Zouhar, Vil{\'e}m and
Navigli, Roberto and
Kocmi, Tom",
editor = "Christodoulopoulos, Christos and
Chakraborty, Tanmoy and
Rose, Carolyn and
Peng, Violet",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
month = nov,
year = "2025",
address = "Suzhou, China",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-emnlp.1317/",
doi = "10.18653/v1/2025.findings-emnlp.1317",
pages = "24261--24285",
ISBN = "979-8-89176-335-7"
}