---
pipeline_tag: translation
language: multilingual
library_name: transformers
base_model:
- FacebookAI/xlm-roberta-large
license: apache-2.0
---
📊 Estimating Machine Translation Difficulty
This repository contains the **SENTINELSRC** metric model used for Difficulty Sampling at the [WMT25 General Machine Translation Shared Task](https://www2.statmt.org/wmt25/translation-task.html), and analyzed in our paper **Estimating Machine Translation Difficulty**.
## Usage
To run this model, install the following git repository:
```bash
pip install git+https://github.com/prosho-97/guardians-mt-eval
```
After that, you can use this model within Python in the following way:
```python
from sentinel_metric import download_model, load_from_checkpoint
model_path = download_model("Prosho/sentinel-src-25")
model = load_from_checkpoint(model_path)
data = [
{"src": "Please sign the form."},
{"src": "He spilled the beans, then backpedaled—talk about mixed signals!"}
]
output = model.predict(data, batch_size=8, gpus=1)
```
Output:
```python
# Segment scores
>>> output.scores
[0.5604351758956909, -0.08413456380367279]
# System score
>>> output.system_score
0.23815030604600906
```
Where the higher the output score, the easier it is to translate the input source text.
## Cite this work
This work has been presented at [EMNLP 2025](https://2025.emnlp.org/). If you use any part, please consider citing our paper as follows:
```bibtex
@inproceedings{proietti-etal-2025-estimating,
title = "Estimating Machine Translation Difficulty",
author = "Proietti, Lorenzo and
Perrella, Stefano and
Zouhar, Vil{\'e}m and
Navigli, Roberto and
Kocmi, Tom",
editor = "Christodoulopoulos, Christos and
Chakraborty, Tanmoy and
Rose, Carolyn and
Peng, Violet",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
month = nov,
year = "2025",
address = "Suzhou, China",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-emnlp.1317/",
doi = "10.18653/v1/2025.findings-emnlp.1317",
pages = "24261--24285",
ISBN = "979-8-89176-335-7"
}
``` 
