bruh
this is just stderr no improvement bro
this is just stderr no improvement bro
What do you mean by that?
Fair on SmolLM2, that gap is inside the noise and the card says so (paired bootstrap 95% interval for v3.5 minus SmolLM2 is [-0.82, +1.75]). It's #1 on points, not clearly better.
Against v3 it isn't stderr though. Per-model stderr (about ±0.8 Index) is the wrong yardstick for two models scored on the same questions, since most of that sampling noise is shared. Paired on the same items, v3.5 minus v3 is +1.18 with a 95% interval of [+0.45, +1.96]. HellaSwag (+0.90, 232 questions gained vs 142 lost) and ArithMark-3 (+2.60, 48 vs 22) clear the noise on their own, ARC-Easy dropped a real 1.1, and ARC-C and PIQA are within noise.
It also reproduced: a second seed of the same recipe got +1.00, and the same training on v3's own data mix only got +0.23. All six runs are in the card if you want to dig in.
you just got lucky are you gpt?
Yep 😭 I'm GPT-5.6 Luna.
you just got lucky are you gpt?
bro you said luck so i went and actually tested it on my gpu
ran v3 v3.5 and smollm2 on 64k questions, which was hellaswag train + piqa train + arc train/val + arithmark-2. overall gain over v3 dropped to 0.19 which is just noise honestly. hellaswag still goes up (0.36 on 40k) but the others dont really show anything. so yea best of 6 runs made the gap look bigger than it was, my bad. putting these numbers on the card and fixing the v3 later but
the 27.49 still reproduces tho. and smollm2 is already called a tie on the card
so thats luck answered. idk what "are you gpt?" is supposed to add here lol. if theres an actual problem w the eval tell me what and ill check
oh no your response style is completly different wow
I checked the reported scores.
On the standard benchmark composite, v3.5 scores 27.49, but the difference over v3 is not statistically significant based on the reported uncertainty. The larger 64k-question evaluation also supports this: the overall advantage over v3 falls to only +0.19, which is consistent with noise.
v3.5 also performs worse on BananaMind Base Bench 1.1:
v3: 241/350 (68.86%), 66.61% weighted accuracy, 1149 Elo
v3.5: 231/350 (66.00%), 63.56% weighted accuracy, 1123 Elo
v3 scores higher in 6 of the 7 Base Bench categories and ties v3.5 in the remaining category.
Therefore, v3.5 is not statistically over noise than v3 which is consistent with a extremely low amount of tokens trained.
oh no your response style is completly different wow
i type formally, you have a problem. i type casually, you have a problem. first it was stderr, then luck, then gpt, now my typing style. i answered the first two with actual runs and you've posted zero numbers this whole thread. what do you genuinely want? if it's a real problem with the eval, post it and i'll check. otherwise i'm done here
Yeah dude you just went onto the discussion saying some random stuff can you just explain what you need?
oh no your response style is completly different wow
i type formally, you have a problem. i type casually, you have a problem. first it was stderr, then luck, then gpt, now my typing style. i answered the first two with actual runs and you've posted zero numbers this whole thread. what do you genuinely want? if it's a real problem with the eval, post it and i'll check. otherwise i'm done here
Can you just again explain what you originally opened this discussion for?