hafsteinn/icelandic-modernbert-ws-pilots

Icelandic ModernBERT warm-start study: learning-rate pilots

Paper: Beyond IceBERT: Open-Data ModernBERT Encoders for Icelandic from 12M to 1B Parameters (arXiv:2610.36230). If you use these models, please cite the paper. The BibTeX entry is at the end of this card.

Nine side experiments of the warm-start study in Beyond IceBERT: Open-Data ModernBERT Encoders for Icelandic from 12M to 1B Parameters, one model per branch (main holds only this page). They test the peak learning rate of warm-started pretraining:

• xs pilot (21 Oct 2026): scratch, Ettin with its vocabulary transplanted (E) and Ettin's layers with new embeddings (Erand), trained straight to 20 tokens per parameter, the warm starts at 1, 0.5 and 0.25 times the scratch peak rate. It set the rate for every warm-started arm of the study (1.0).
• Review check C5 (24 Oct 2026): Ettin and mmBERT-keep at base, trained straight to 10 tokens per parameter at half the rate, against the study's 10-token branches. Half the rate is no better on any task.

branch	arm	size	peak LR	tokens/param	MLM loss at the end
wsp-S-xs-s0-b20	Scratch	xs	0.002 (1 x scratch)	20	2.756
wsp-E-xs-s0-b20	Ettin	xs	0.002 (1 x scratch)	20	2.321
wsp-E-xs-s0-lr0.5-b20	Ettin	xs	0.001 (0.5 x scratch)	20	2.368
wsp-E-xs-s0-lr0.25-b20	Ettin	xs	0.0005 (0.25 x scratch)	20	2.520
wsp-Erand-xs-s0-b20	Ettin-body	xs	0.002 (1 x scratch)	20	2.443
wsp-Erand-xs-s0-lr0.5-b20	Ettin-body	xs	0.001 (0.5 x scratch)	20	2.512
wsp-Erand-xs-s0-lr0.25-b20	Ettin-body	xs	0.0005 (0.25 x scratch)	20	2.726
wsp-E-base-s0-lr0.5-b10	Ettin	base	0.0004 (0.5 x scratch)	10	1.557
wsp-Mkeep-base-s0-lr0.5-b10	mmBERT-keep	base	0.0004 (0.5 x scratch)	10	0.440

MLM loss: validation, 15% masking, mean over eight sources; mmBERT-keep's is per Gemma token. Each branch's card gives the recipe, the comparison and the caveats.

from transformers import AutoModelForMaskedLM
m = AutoModelForMaskedLM.from_pretrained("hafsteinn/icelandic-modernbert-ws-pilots", revision="wsp-E-base-s0-lr0.5-b10")

Fill-mask. For every branch but wsp-Mkeep-base-s0-lr0.5-b10, write the mask token right after the previous word, with no space (Reykjavík er höfuðborg[MASK].); the branch cards say why.

These are not the models to use; the size ladder and the study's curves are in the collection https://huggingface.co/collections/hafsteinn/icelandic-modernbert-6adbbfe167c872dddf40ce2a.

Full numbers: the paper, arXiv:2610.36230.

Citation

@misc{einarsson2026icebertopendatamodernbertencoders,
  title         = {Beyond {IceBERT}: Open-Data {ModernBERT} Encoders for {Icelandic} from {12M} to {1B} Parameters},
  author        = {Hafsteinn Einarsson},
  year          = {2026},
  eprint        = {2610.36230},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2610.36230}
}