How to use from
llama.cpp
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0
# Run inference directly in the terminal:
llama cli -hf precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0
Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0
# Run inference directly in the terminal:
llama cli -hf precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0
# Run inference directly in the terminal:
./build/bin/llama cli -hf precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0
Use Docker
docker model run hf.co/precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0
Quick Links

Precision Numerics logo

Boole-32B-A2.5B-Preview-GGUF

Base model: Flint  |  Code: boole-demo  |  llama.cpp fork  |  Apache 2.0

1. Model Introduction

Boole-32B-A2.5B-Preview is Flint compressed to 9.0 GB: 21% of its routed experts are pruned and the rest average 1.8 bits per weight. It decodes 2.0x faster than the fastest Flint GGUF on one RTX 5090.

2. Model Summary

ArchitectureMixture-of-Experts
Total / activated parameters32B / 2.5B
Routed experts9,541 (2,491 of Flint's 12,032 pruned)
Routed expert precision1, 2 or 4 bits per expert; 1.8 bits per weight on average
Other weights4-bit (router BF16)
File size9.0 GB (GGUF)

3. Evaluation Results

Score vs. model size on GSM8K, IFEval, IFBench, LiveCodeBench v6, HumanEval+ and MBPP+: Boole-32B-A2.5B-Preview (9.0 GB) against Unsloth UD-IQ1_M (11.4 GB), Flint INT4 (26.0 GB) and Flint BF16 (79.6 GB)
BenchmarkBoole-32B-A2.5B-PreviewUnsloth UD-IQ1_MFlint INT4Flint BF16
Size9.0 GB11.4 GB26.0 GB79.6 GB
GSM8K93.2 ยฑ 0.394.9 ยฑ 0.195.8 ยฑ <0.196.1 ยฑ <0.1
IFEval (prompt-strict)90.6 ยฑ 0.193.5 ยฑ 0.294.9 ยฑ 0.294.8 ยฑ 0.2
IFBench (prompt-strict)75.9 ยฑ 1.582.6 ยฑ 0.587.7 ยฑ 0.689.2 ยฑ 0.4
LiveCodeBench v667.3 ยฑ 1.177.2 ยฑ 0.581.3 ยฑ 1.081.9 ยฑ 0.3
HumanEval+90.0 ยฑ 1.290.5 ยฑ 1.092.5 ยฑ 0.792.7 ยฑ 0.9
MBPP+76.7 ยฑ 0.378.5 ยฑ 0.778.9 ยฑ 0.779.8 ยฑ 0.2

Mean ยฑ standard error. Temperature 1.0, top-p 0.95, top-k 20, 65,536-token context. LiveCodeBench v6: 454 problems, Aug 2024 โ€“ May 2025.

4. Inference Speed

Total throughput vs. decode speed per user on one RTX 5090, 8K prompt / 1K output, 1 to 32 concurrent requests: Boole-32B-A2.5B-Preview against Unsloth UD-IQ1_M and Flint INT4 llama-bench on one RTX 5090: Boole-32B-A2.5B-Preview is 1.61x faster at prompt processing and 3.53x faster at decode than the fastest baseline
1ร— RTX 5090Boole-32B-A2.5B-PreviewUnsloth UD-IQ1_MFlint INT4
Prompt processing
pp512 (tok/s)
19,371 ยฑ 25711,882 ยฑ 812,013 ยฑ 18
Decode
tg128 (tok/s)
987.0 ยฑ 0.6264.9 ยฑ 0.2279.4 ยฑ 0.3
Time to first token
8K prompt (s)
0.450.610.62
Peak throughput
8K prompt / 1K output (tok/s)
10,337 at 327,216 at 325,755 at 16

llama-bench rows: mean ยฑ standard deviation. Peak throughput: prompt + output tokens/s, at the number of concurrent requests shown. Baselines run on upstream llama.cpp and Boole on our llama.cpp fork; all models use -fa on -ub 1024 and GGML_CUDA_GRAPH_OPT=1.

5. What Our llama.cpp Fork Changes

(a) Experts regrouped by bit width in the GGUF (layer 22: 61 at 1-bit, 111 at 2-bit, 31 at 4-bit, 53 pruned). (b) One MoE layer's decode step: new RMSNorm kernels that also write an 8-bit copy, fused mixed 1/2/4-bit expert kernels; changed QKV projection, flash attention and router top-k

6. Quick Start

Use boole-demo, which builds our llama.cpp fork at a pinned commit (CUDA GPU required; stock llama.cpp cannot load this model yet):

git clone https://github.com/PrecisionNumerics/boole-demo.git && cd boole-demo
bash scripts/setup_llama_cpp.sh
hf download precision-numerics/Boole-32B-A2.5B-Preview-GGUF Boole-32B-A2.5B-Preview-Q2_0.gguf --local-dir model
GGML_CUDA_GRAPH_OPT=1 third_party/llama.cpp/build/bin/llama-server -m model/Boole-32B-A2.5B-Preview-Q2_0.gguf -ngl 99 -fa on -ub 1024 -c 32768

7. License

Apache 2.0. NOTICE credits the upstream Flint model.

8. Citation

@misc{precisionnumerics2026boole32ba2p5b,
  author = {{Precision Numerics}},
  title = {{Boole}-32B-A2.5B-Preview},
  year = {2026},
  month = {November},
  url = {https://huggingface.co/precision-numerics/Boole-32B-A2.5B-Preview-GGUF}
}
Downloads last month
1,391
GGUF
Model size
32B params
Architecture
flint
Hardware compatibility
Log In to add your hardware

2-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for precision-numerics/Boole-32B-A2.5B-Preview-GGUF

Quantized
(110)
this model

Collection including precision-numerics/Boole-32B-A2.5B-Preview-GGUF