Instructions to use nota-ai/Solar-Open-100B-Nota-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nota-ai/Solar-Open-100B-Nota-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="nota-ai/Solar-Open-100B-Nota-FP8", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("nota-ai/Solar-Open-100B-Nota-FP8", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("nota-ai/Solar-Open-100B-Nota-FP8", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use nota-ai/Solar-Open-100B-Nota-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nota-ai/Solar-Open-100B-Nota-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nota-ai/Solar-Open-100B-Nota-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/nota-ai/Solar-Open-100B-Nota-FP8

SGLang

How to use nota-ai/Solar-Open-100B-Nota-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nota-ai/Solar-Open-100B-Nota-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nota-ai/Solar-Open-100B-Nota-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nota-ai/Solar-Open-100B-Nota-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nota-ai/Solar-Open-100B-Nota-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use nota-ai/Solar-Open-100B-Nota-FP8 with Docker Model Runner:
```
docker model run hf.co/nota-ai/Solar-Open-100B-Nota-FP8
```

Please make nvfp4 quant and make mlx version of that quant

by Narutoouz - opened Feb 1

Discussion

Narutoouz

Feb 1

thanks for making the fp8 quant.

SangminLee-NOTA

Nota AI org Feb 2

@Narutoouz

Thanks for the suggestion!

We have some good news regarding nvfp4 — it is currently work-in-progress internally, so please stay tuned for the upcoming update!
As for the mlx version, we don't have immediate plans at the moment, but we've passed your request to our team to review its feasibility for future releases.

Thanks again for your interest in our work!

hancheolp

Nota AI org Mar 11

•

edited Mar 11

@Narutoouz

A Solar-Open NVFP4 version quantized using NotaMoEQuant is now available.
https://huggingface.co/nota-ai/Solar-Open-100B-NotaMoEQuant-NVFP4

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment