deberta-v3-base-unified-mcqa-2-choice

If finetuning this further on downstream tasks (classification, etc) you may need to pass ignore_mismatched_sizes=True when loading this model.

This model is a fine-tuned version of microsoft/deberta-v3-base on the pszemraj/unified-mcqa dataset. It achieves the following results on the evaluation set:

• Loss: 0.3384
• Accuracy: 0.9010
• Num Input Tokens Seen: 15919768

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 69
• gradient_accumulation_steps: 8
• total_train_batch_size: 64
• optimizer: Use adamw_torch_fused with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 300
• num_epochs: 3.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Input Tokens Seen
0.3639	0.4579	400	0.3093	0.8700	2444384
0.308	0.9157	800	0.2792	0.8920	4861008
0.2517	1.3732	1200	0.2989	0.8890	7293776
0.2399	1.8310	1600	0.2796	0.8990	9723744
0.1082	2.2885	2000	0.3557	0.9030	12157608
0.1269	2.7463	2400	0.3486	0.8980	14592584

Framework versions

• Transformers 4.51.3
• Pytorch 2.6.0+cu124
• Datasets 3.5.0
• Tokenizers 0.21.1