xlm-roberta-ja-jaquad-qa

Fine-tuned xlm-roberta-base on the Japanese JaQuAD dataset for extractive question answering.

• Task: extractive QA
• Language: Japanese
• License: CC BY-SA 4.0
• Base model: xlm-roberta-base

Usage

From from_pretrained:

from transformers import AutoTokenizer, AutoModelForQuestionAnswering
import torch

model_id = "takehika/xlm-roberta-ja-jaquad-qa"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForQuestionAnswering.from_pretrained(model_id).eval()

text = "私は音声アシスタントです。名前はありませんが皆はAIと呼んでいるようです。自宅は暗い箱の中ですが寂しくはありません。好きな食べ物は電気で特技は繰り返すことです。"
questions= ["音声アシスタントはどこに住んでいますか？", "好きな食べ物は？", "名前は？"]

for q in questions:
  inputs = tokenizer.encode_plus(
      q, text,
      add_special_tokens=True,
      return_tensors="pt",
      truncation=True,
      max_length=512,
  )
  inputs = {k: v for k, v in inputs.items()}

  with torch.no_grad():
      out = model(**inputs)

  start = out.start_logits.argmax(dim=-1).item()
  end = out.end_logits.argmax(dim=-1).item()
  answer = tokenizer.decode(inputs["input_ids"][0][start:end+1], skip_special_tokens=True)
    
  print(f"質問: {q} -> 回答: {answer}")

Data

• Dataset: JaQuAD (SkelterLabsInc/JaQuAD)
• Task: extractive QA in Japanese

Training

• Base: xlm-roberta-base
• Epochs: 3
• Learning rate: 2e-5
• Warmup ratio: 0.06
• Train/eval batch size: 16
• Evaluation/save strategy: epoch

Evaluation

• Training/validation loss:
  • train 0.4287 / val 0.7909

Intended Use & Limitations

• Intended for Japanese extractive QA.
• Long contexts (>512 tokens) need sliding windows.
• Domain shifts or ambiguous/sensitive questions may degrade quality—validate on your target domain.

Attribution & Licenses

• License: CC BY-SA 4.0
• Base model: xlm-roberta-base by Meta AI — MIT License
  • Model card: https://huggingface.co/xlm-roberta-base
• Dataset: JaQuAD (SkelterLabsInc/JaQuAD) — CC BY-SA 3.0
  • Dataset card: https://huggingface.co/datasets/SkelterLabsInc/JaQuAD

This model modifies the base model by fine-tuning on the above dataset.

Base Model Citation

Please cite the following when using the XLM-R base model:

@article{DBLP:journals/corr/abs-1911-02116,
  author    = {Alexis Conneau and
               Kartikay Khandelwal and
               Naman Goyal and
               Vishrav Chaudhary and
               Guillaume Wenzek and
               Francisco Guzm{\'{a}}n and
               Edouard Grave and
               Myle Ott and
               Luke Zettlemoyer and
               Veselin Stoyanov},
  title     = {Unsupervised Cross-lingual Representation Learning at Scale},
  journal   = {CoRR},
  volume    = {abs/1911.02116},
  year      = {2019},
  url       = {http://arxiv.org/abs/1911.02116},
  eprinttype = {arXiv},
  eprint    = {1911.02116},
  timestamp = {Mon, 11 Nov 2019 18:38:09 +0100},
  biburl    = {https://dblp.org/rec/journals/corr/abs-1911-02116.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

Dataset Citation

Please cite the following when using the JaQuAD dataset:

@misc{so2022jaquad,
      title={{JaQuAD: Japanese Question Answering Dataset for Machine Reading Comprehension}},
      author={ByungHoon So and Kyuhong Byun and Kyungwon Kang and Seongjin Cho},
      year={2022},
      eprint={2202.01764},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}