What's the point of this quantization?
- deepseek-ai/DeepSeek-V4-Flash: 160 GB
- nvidia/DeepSeek-V4-Flash-NVFP4: 168 GB
Why would I need a quantized model worse than original and even larger?
What the hell am I gonna do with a nvfp4 model? it's not even sparse. Is it actually faster? Any data?
maybe compatibility? I've heard about problems with the original model running on an RTX 6000, for example. But that's just a guess… I thought the same as you.
I tested both this vs the hybrid deepseek. dual rtx6000 pro, pcie 3.0:
Test NVIDIA NVFP4 Official hybrid Difference
━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━━━━
64K prompt TTFT, c1 12.94s 13.39s NVIDIA 3.5% faster
─────────────────────── ────────────── ───────────────── ──────────────────────
64K prompt TTFT, c2 19.63s 20.41s NVIDIA 4.0% faster
─────────────────────── ────────────── ───────────────── ──────────────────────
128K prompt TTFT, c1 26.23s 26.96s NVIDIA 2.8% faster
─────────────────────── ────────────── ───────────────── ──────────────────────
128K prompt TTFT, c2 39.01s 40.41s NVIDIA 3.6% faster
─────────────────────── ────────────── ───────────────── ──────────────────────
64K + 512 output, c1 30.04 tok/s 30.82 tok/s Official 2.6% faster
─────────────────────── ────────────── ───────────────── ──────────────────────
64K + 512 output, c2 33.43 tok/s 33.06 tok/s Essentially tied
─────────────────────── ────────────── ───────────────── ──────────────────────
128K + 512 output, c1 17.11 tok/s 17.01 tok/s Essentially tied
─────────────────────── ────────────── ───────────────── ──────────────────────
128K + 512 output, c2 18.38 tok/s 17.48 tok/s NVIDIA 5.1% faster
─────────────────────── ────────────── ───────────────── ──────────────────────
64K + 2048 output, c1 75.17 tok/s 78.99 tok/s Official 5.1% faster
─────────────────────── ────────────── ───────────────── ──────────────────────
64K + 2048 output, c2 92.68 tok/s 97.89 tok/s Official 5.6% faster
It would be awesome though if nvidia did a dspark version of this repo
So the speed gain from NVFP4 is not significant. It also runs the risk of weaker intelligence. Therefore, is it better to stick with the original as the memory footprint is slightly less and also support DSpark?
So the speed gain from NVFP4 is not significant. It also runs the risk of weaker intelligence. Therefore, is it better to stick with the original as the memory footprint is slightly less and also support DSpark?
I am definitely sticking with original dspark model. No reason to risk lossing precision in most layers for pathetic 5% boost.
Is it theoretically possible for Nvidia to pay DeepSeek such that they can work together to make a NVFP4 version that has DSpark?