Instructions to use ethzanalytics/dolly-v2-12b-sharded-8bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ethzanalytics/dolly-v2-12b-sharded-8bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ethzanalytics/dolly-v2-12b-sharded-8bit")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ethzanalytics/dolly-v2-12b-sharded-8bit")
model = AutoModelForCausalLM.from_pretrained("ethzanalytics/dolly-v2-12b-sharded-8bit")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use ethzanalytics/dolly-v2-12b-sharded-8bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ethzanalytics/dolly-v2-12b-sharded-8bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ethzanalytics/dolly-v2-12b-sharded-8bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/ethzanalytics/dolly-v2-12b-sharded-8bit

SGLang

How to use ethzanalytics/dolly-v2-12b-sharded-8bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ethzanalytics/dolly-v2-12b-sharded-8bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ethzanalytics/dolly-v2-12b-sharded-8bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ethzanalytics/dolly-v2-12b-sharded-8bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ethzanalytics/dolly-v2-12b-sharded-8bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use ethzanalytics/dolly-v2-12b-sharded-8bit with Docker Model Runner:
```
docker model run hf.co/ethzanalytics/dolly-v2-12b-sharded-8bit
```

dolly-v2-12b: sharded 8bit checkpoint

This is a sharded checkpoint (with ~4GB shards) of the databricks/dolly-v2-12b model in 8bit precision using bitsandbytes.

Refer to the original model for all details w.r.t. to the model. For more info on loading 8bit models, refer to the example repo and/or the 4.28.0 release info.

total model size is only ~12.5 GB!
this enables low-RAM loading, i.e. Colab :)
update: generation speed can be greatly improved by setting use_cache=True and generating via contrastive search. example notenook here

Basic Usage

install/upgrade transformers, accelerate, and bitsandbytes. For this to work you must have transformers>=4.28.0 and bitsandbytes>0.37.2.

pip install -U -q transformers bitsandbytes accelerate

Load the model. As it is serialized in 8bit you don't need to do anything special:

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "ethzanalytics/dolly-v2-12b-sharded-8bit"
tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(model_name)

Downloads last month: 7

ethzanalytics
/

dolly-v2-12b-sharded-8bit

dolly-v2-12b: sharded 8bit checkpoint

Basic Usage

Dataset used to train ethzanalytics/dolly-v2-12b-sharded-8bit