Excellent !
#6
by DavidAU - opened
Excellent work.
Both want to try the model and try this in fine tuning.
It would be interesting to see how it works with Gemma 4 "E2B" and "E4B" versions.
Qwens too.
Especially Qwens - they so need "better prose" .
@kabachuha
No promises ; it is touch and go with lm_head only tuning so far.
Generally I leave it alone or tune with the rest of the tensors.