Qwen3-4B-GiantMIDI-REMI

Qwen3-4B adapted to symbolic piano generation on GiantMIDI-Piano, with the LoRA adapters and the 316 new REMI music-token embeddings merged into the full weights. This is the "out-of-domain pretraining" arm of a CSE 253 (UCSD) study on whether pretraining domain or scale matters more for symbolic music.

What was trained

The text base never saw music. We added the 316 REMI tokens (<MIDI_TOK_0..315>) as special tokens, froze the base in 4-bit, and trained LoRA (r=16) plus only the 316 new embedding rows. The full bf16 weights here are the merged result.

Usage

The tokenizer includes the 316 <MIDI_TOK_i> tokens, each mapping 1:1 to a MidiTok REMI id (316-token REMI vocabulary). Generation is constrained to those tokens, then decoded to MIDI with MidiTok.

Results (test set, REMI vocabulary)

Perplexity 13.55, pitch KL 0.49, interval KL 0.51, repetition 0.022, 0% invalid.

Base model: Qwen/Qwen3-4B (Apache-2.0). Coursework artifact.