Transformers
xlm-roberta
🇪🇺 Region: EU
AbhayShukla19 commited on
Commit
02df839
·
verified ·
1 Parent(s): bd55a5e

Call post_init() in XLMRobertaLoRA for transformers v5

Browse files

Fixes #62. Heads up: I prepared this with AI assistance (Claude Code) and reviewed and tested it myself.

`XLMRobertaLoRA.__init__` doesn't call `self.post_init()`, which transformers v5 requires (#60 added it to the classes in `modeling_xlm_roberta.py`, but not here). So loading `jinaai/jina-embeddings-v3` on transformers>=5 fails with:

```
AttributeError: 'XLMRobertaLoRA' object has no attribute 'all_tied_weights_keys'
```

This adds `self.post_init()` at the end of `XLMRobertaLoRA.__init__`.

**Testing** (RTX PRO 6000; `SentenceTransformer("jinaai/jina-embeddings-v3", trust_remote_code=True).encode(..., task="text-matching")` on 4 sentences)

| transformers | current code | this PR |
|---|---|---|
| 4.57.6 | works | embeddings bit-identical to current code |
| 5.17.0 | `AttributeError` above | embeddings bit-identical to 4.57.6 |

- The `load_trained_adapters=False` path (fresh adapters on a pretrained `XLMRobertaModel`) gives a bit-identical state dict to the current code on 4.57.6 (seeded, 492/492 tensors equal), so `post_init()` doesn't re-initialize anything there.
- vLLM's `test_matryoshka` HF-vs-vLLM comparison for jina-embeddings-v3 passes with this change on transformers 5.17.0 (it's currently skipped in vLLM because of this error).

Files changed (1) hide show
  1. modeling_lora.py +1 -0
modeling_lora.py CHANGED
@@ -290,6 +290,7 @@ class XLMRobertaLoRA(XLMRobertaPreTrainedModel):
290
  alpha=self._alpha,
291
  )
292
  self.main_params_trainable = config.lora_main_params_trainable
 
293
 
294
  @property
295
  def rotary_emb_base(self):
 
290
  alpha=self._alpha,
291
  )
292
  self.main_params_trainable = config.lora_main_params_trainable
293
+ self.post_init()
294
 
295
  @property
296
  def rotary_emb_base(self):