Instructions to use precision-numerics/Boole-32B-A2.5B-Preview-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use precision-numerics/Boole-32B-A2.5B-Preview-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0 # Run inference directly in the terminal: llama cli -hf precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0 # Run inference directly in the terminal: llama cli -hf precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0 # Run inference directly in the terminal: ./build/bin/llama cli -hf precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0
Use Docker
docker model run hf.co/precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0
- Llama new
- LM Studio
- Jan
- vLLM
How to use precision-numerics/Boole-32B-A2.5B-Preview-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "precision-numerics/Boole-32B-A2.5B-Preview-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "precision-numerics/Boole-32B-A2.5B-Preview-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0
- Ollama
How to use precision-numerics/Boole-32B-A2.5B-Preview-GGUF with Ollama:
ollama run hf.co/precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0
- Unsloth Desktop
- Pi
How to use precision-numerics/Boole-32B-A2.5B-Preview-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use precision-numerics/Boole-32B-A2.5B-Preview-GGUF with Docker Model Runner:
docker model run hf.co/precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0
- Lemonade
How to use precision-numerics/Boole-32B-A2.5B-Preview-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0
Run and chat with the model
lemonade run user.Boole-32B-A2.5B-Preview-GGUF-Q2_0
List all available models
lemonade list
- Hermes Agent
How to use precision-numerics/Boole-32B-A2.5B-Preview-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use precision-numerics/Boole-32B-A2.5B-Preview-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "precision-numerics/Boole-32B-A2.5B-Preview-GGUF:Q2_0" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Boole-32B-A2.5B-Preview-GGUF
Base model: Flint | Code: boole-demo | llama.cpp fork | Apache 2.0
1. Model Introduction
Boole-32B-A2.5B-Preview is Flint compressed to 9.0 GB: 21% of its routed experts are pruned and the rest average 1.8 bits per weight. It decodes 2.0x faster than the fastest Flint GGUF on one RTX 5090.
2. Model Summary
| Architecture | Mixture-of-Experts |
| Total / activated parameters | 32B / 2.5B |
| Routed experts | 9,541 (2,491 of Flint's 12,032 pruned) |
| Routed expert precision | 1, 2 or 4 bits per expert; 1.8 bits per weight on average |
| Other weights | 4-bit (router BF16) |
| File size | 9.0 GB (GGUF) |
3. Evaluation Results
| Benchmark | Boole-32B-A2.5B-Preview | Unsloth UD-IQ1_M | Flint INT4 | Flint BF16 |
|---|---|---|---|---|
| Size | 9.0 GB | 11.4 GB | 26.0 GB | 79.6 GB |
| GSM8K | 93.2 ยฑ 0.3 | 94.9 ยฑ 0.1 | 95.8 ยฑ <0.1 | 96.1 ยฑ <0.1 |
| IFEval (prompt-strict) | 90.6 ยฑ 0.1 | 93.5 ยฑ 0.2 | 94.9 ยฑ 0.2 | 94.8 ยฑ 0.2 |
| IFBench (prompt-strict) | 75.9 ยฑ 1.5 | 82.6 ยฑ 0.5 | 87.7 ยฑ 0.6 | 89.2 ยฑ 0.4 |
| LiveCodeBench v6 | 67.3 ยฑ 1.1 | 77.2 ยฑ 0.5 | 81.3 ยฑ 1.0 | 81.9 ยฑ 0.3 |
| HumanEval+ | 90.0 ยฑ 1.2 | 90.5 ยฑ 1.0 | 92.5 ยฑ 0.7 | 92.7 ยฑ 0.9 |
| MBPP+ | 76.7 ยฑ 0.3 | 78.5 ยฑ 0.7 | 78.9 ยฑ 0.7 | 79.8 ยฑ 0.2 |
Mean ยฑ standard error. Temperature 1.0, top-p 0.95, top-k 20, 65,536-token context. LiveCodeBench v6: 454 problems, Aug 2024 โ May 2025.
4. Inference Speed
| 1ร RTX 5090 | Boole-32B-A2.5B-Preview | Unsloth UD-IQ1_M | Flint INT4 |
|---|---|---|---|
| Prompt processing pp512 (tok/s) | 19,371 ยฑ 257 | 11,882 ยฑ 8 | 12,013 ยฑ 18 |
| Decode tg128 (tok/s) | 987.0 ยฑ 0.6 | 264.9 ยฑ 0.2 | 279.4 ยฑ 0.3 |
| Time to first token 8K prompt (s) | 0.45 | 0.61 | 0.62 |
| Peak throughput 8K prompt / 1K output (tok/s) | 10,337 at 32 | 7,216 at 32 | 5,755 at 16 |
llama-bench rows: mean ยฑ standard deviation. Peak throughput: prompt + output tokens/s, at the number of concurrent requests shown. Baselines run on upstream llama.cpp and Boole on our llama.cpp fork; all models use -fa on -ub 1024 and GGML_CUDA_GRAPH_OPT=1.
5. What Our llama.cpp Fork Changes
6. Quick Start
Use boole-demo, which builds our llama.cpp fork at a pinned commit (CUDA GPU required; stock llama.cpp cannot load this model yet):
git clone https://github.com/PrecisionNumerics/boole-demo.git && cd boole-demo
bash scripts/setup_llama_cpp.sh
hf download precision-numerics/Boole-32B-A2.5B-Preview-GGUF Boole-32B-A2.5B-Preview-Q2_0.gguf --local-dir model
GGML_CUDA_GRAPH_OPT=1 third_party/llama.cpp/build/bin/llama-server -m model/Boole-32B-A2.5B-Preview-Q2_0.gguf -ngl 99 -fa on -ub 1024 -c 32768
7. License
Apache 2.0. NOTICE credits the upstream Flint model.
8. Citation
@misc{precisionnumerics2026boole32ba2p5b,
author = {{Precision Numerics}},
title = {{Boole}-32B-A2.5B-Preview},
year = {2026},
month = {November},
url = {https://huggingface.co/precision-numerics/Boole-32B-A2.5B-Preview-GGUF}
}
- Downloads last month
- 1,391
2-bit
Model tree for precision-numerics/Boole-32B-A2.5B-Preview-GGUF
Base model
RecursiveSI/flint