Instructions to use OpenAssistant/llama2-13b-orca-8k-3319 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use OpenAssistant/llama2-13b-orca-8k-3319 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="OpenAssistant/llama2-13b-orca-8k-3319")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("OpenAssistant/llama2-13b-orca-8k-3319")
model = AutoModelForCausalLM.from_pretrained("OpenAssistant/llama2-13b-orca-8k-3319", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use OpenAssistant/llama2-13b-orca-8k-3319 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "OpenAssistant/llama2-13b-orca-8k-3319"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenAssistant/llama2-13b-orca-8k-3319",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/OpenAssistant/llama2-13b-orca-8k-3319

SGLang

How to use OpenAssistant/llama2-13b-orca-8k-3319 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "OpenAssistant/llama2-13b-orca-8k-3319" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenAssistant/llama2-13b-orca-8k-3319",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "OpenAssistant/llama2-13b-orca-8k-3319" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenAssistant/llama2-13b-orca-8k-3319",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use OpenAssistant/llama2-13b-orca-8k-3319 with Docker Model Runner:
```
docker model run hf.co/OpenAssistant/llama2-13b-orca-8k-3319
```

example code is not working

by Q4234 - opened Jul 26, 2023

Discussion

Q4234

Jul 26, 2023

File /usr/local/lib/python3.9/dist-packages/accelerate/hooks.py:165, in add_hook_to_module..new_forward(*args, **kwargs)
163 output = old_forward(*args, **kwargs)
164 else:
--> 165 output = old_forward(*args, **kwargs)
166 return module._hf_hook.post_forward(module, output)

File /usr/local/lib/python3.9/dist-packages/transformers/models/llama/modeling_llama.py:300, in LlamaAttention.forward(self, hidden_states, attention_mask, position_ids, past_key_value, output_attentions, use_cache)
297 key_slices = self.k_proj.weight.split(key_value_slicing, dim=0)
298 value_slices = self.v_proj.weight.split(key_value_slicing, dim=0)
--> 300 query_states = [F.linear(hidden_states, query_slices[i]) for i in range(self.config.pretraining_tp)]
301 query_states = torch.cat(query_states, dim=-1)
303 key_states = [F.linear(hidden_states, key_slices[i]) for i in range(self.config.pretraining_tp)]

File /usr/local/lib/python3.9/dist-packages/transformers/models/llama/modeling_llama.py:300, in (.0)
297 key_slices = self.k_proj.weight.split(key_value_slicing, dim=0)
298 value_slices = self.v_proj.weight.split(key_value_slicing, dim=0)
--> 300 query_states = [F.linear(hidden_states, query_slices[i]) for i in range(self.config.pretraining_tp)]
301 query_states = torch.cat(query_states, dim=-1)
303 key_states = [F.linear(hidden_states, key_slices[i]) for i in range(self.config.pretraining_tp)]

RuntimeError: mat1 and mat2 shapes cannot be multiplied (69x5120 and 1x2560)

nigh8w0lf

Jul 26, 2023

Got the same shape mismatch error , using latest transformers>=4.31.0

Q4234

Jul 26, 2023

I guess setting the ctxlen is missing in the code ... no idea how to do this but the code for SuperCOT/SuperHOT did do this...

ttronrud

Jul 26, 2023

They need to set pretraining_tp to 1 for it to work with quantized models. You can set the value in the local copy of the model's config and that'll fix the problem.
https://github.com/facebookresearch/llama/issues/423#issuecomment-1643387661

andreaskoepf

OpenAssistant org Jul 27, 2023

@ttronrud thanks for reporting. We added pretraining_tp to the config.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment