Skip to main content

Models

Below is a list of popular OSS models that you can query instantly or deploy on dedicated hardware with Predibase. The models are available via our UI Playground, Python SDK, or REST API.

Shared Serverless

Deployment NameParametersArchitectureLicenseContext WindowAlways On
llama-3-1-8b8 billionLlama-3Meta (request for commercial use)63999No
llama-3-1-8b-instruct8 billionLlama-3Meta (request for commercial use)63999✅ Yes
llama-3-8b8 billionLlama-3Meta (request for commercial use)8192No
llama-3-8b-instruct8 billionLlama-3Meta (request for commercial use)8192No
llama-3-70b70 billionLlama-3Meta (request for commercial use)8192No
llama-3-70b-instruct70 billionLlama-3Meta (request for commercial use)8192No
llama-2-7b7 billionLlama-2Meta (request for commercial use)4096No
llama-2-7b-chat7 billionLlama-2Meta (request for commercial use)4096No
llama-2-13b13 billionLlama-2Meta (request for commercial use)4096No
llama-2-13b-chat13 billionLlama-2Meta (request for commercial use)4096No
llama-2-70b70 billionLlama-2Meta (request for commercial use)4096No
llama-2-70b-chat70 billionLlama-2Meta (request for commercial use)4096No
codellama-7b7 billionLlama-2Meta (request for commercial use)4096No
codellama-7b-instruct7 billionLlama-2Meta (request for commercial use)4096No
codellama-13b-instruct13 billionLlama-2Meta (request for commercial use)4096No
codellama-70b-instruct70 billionLlama-2Meta (request for commercial use)4096No
mistral-7b7 billionMistralApache 2.08000No
mistral-7b-instruct7 billionMistralApache 2.08000No
mistral-7b-instruct-v0-27 billionMistralApache 2.08000✅ Yes
mistral-7b-instruct-v0-37 billionMistralApache 2.08000No
mixtral-8x7b-v0-146.7 billionMixtralApache 2.032768No
mixtral-8x7b-instruct-v0-146.7 billionMixtralApache 2.032768No
solar-1-mini-chat-24061210.7 billionSolarCustom License32768✅ Yes
zephyr-7b-beta7 billionMistralMIT8000No
phi-22.7 billionPhi-2MIT2048No
phi-3-mini-4k-instruct3.8 billionPhi-3MIT4096No
gemma-2b2.5 billionGemmaGoogle8192No
gemma-2b-instruct2.5 billionGemmaGoogle8192No
gemma-7b8.5 billionGemmaGoogle8192No
gemma-7b-instruct8.5 billionGemmaGoogle8192No
gemma-2-9b9.24 billionGemmaGoogle8192No
gemma-2-9b-instruct9.24 billionGemmaGoogle8192No
gemma-2-27b27.2 billionGemmaGoogle8192No
gemma-2-27b-instruct27.2 billionGemmaGoogle8192No
qwen2-7b7.62 billionQwenTongyi Qianwen131072No
qwen2-7b-instruct7.62 billionQwenTongyi Qianwen131072No
qwen2-72b72.7 billionQwenTongyi Qianwen131072No
qwen2-72b-instruct72.7 billionQwenTongyi Qianwen131072No
info

Note: Models that are not always on scale down to 0 and may have a brief spin up time before serving requests. If you would like us to add support for any shared serverless endpoints or make any existing endpoints always on, please get in touch on Discord.

Private Serverless

When you're ready for production, Predibase also offers the ability to spin up private instances of nearly any open-source model available. These models fall into two categories:

  1. Available LLMs: These are models we have first-class support for. These have been verified and are ensured to work well.
  2. Best-Effort LLMs: These are models that have not been verified and may occasionally not deploy as expected.

Available LLMs

Deployment NameParametersArchitectureLicenseContext Window (Max Tokens)
llama-3-1-8b8 billionLlama-3Meta (request for commercial use)63999
llama-3-1-8b-instruct8 billionLlama-3Meta (request for commercial use)63999
llama-3-8b8 billionLlama-3Meta (request for commercial use)8192
llama-3-8b-instruct8 billionLlama-3Meta (request for commercial use)8192
llama-3-70b70 billionLlama-3Meta (request for commercial use)8192
llama-3-70b-instruct70 billionLlama-3Meta (request for commercial use)8192
llama-2-7b7 billionLlama-2Meta (request for commercial use)4096
llama-2-7b-chat7 billionLlama-2Meta (request for commercial use)4096
llama-2-13b13 billionLlama-2Meta (request for commercial use)4096
llama-2-13b-chat13 billionLlama-2Meta (request for commercial use)4096
llama-2-70b70 billionLlama-2Meta (request for commercial use)4096
llama-2-70b-chat70 billionLlama-2Meta (request for commercial use)4096
codellama-7b7 billionLlama-2Meta (request for commercial use)4096
codellama-7b-instruct7 billionLlama-2Meta (request for commercial use)4096
codellama-13b-instruct13 billionLlama-2Meta (request for commercial use)4096
codellama-70b-instruct70 billionLlama-2Meta (request for commercial use)4096
mistral-7b7 billionMistralApache 2.032768
mistral-7b-instruct7 billionMistralApache 2.032768
mistral-7b-instruct-v0-27 billionMistralApache 2.032768
mistral-7b-instruct-v0-37 billionMistralApache 2.032768
mixtral-8x7b-v0-146.7 billionMixtralApache 2.032768
mixtral-8x7b-instruct-v0-146.7 billionMixtralApache 2.032768
solar-1-mini-chat-24061210.7 billionSolarCustom License32768
zephyr-7b-beta7 billionMistralMIT8000
phi-22.7 billionPhi-2MIT2048
phi-3-mini-4k-instruct3.8 billionPhi-3MIT4096
gemma-2b2.5 billionGemmaGoogle8192
gemma-2b-instruct2.5 billionGemmaGoogle8192
gemma-7b*8.5 billionGemmaGoogle8192
gemma-7b-instruct*8.5 billionGemmaGoogle8192
gemma-2-9b9.24 billionGemmaGoogle8192
gemma-2-9b-instruct9.24 billionGemmaGoogle8192
gemma-2-27b27.2 billionGemmaGoogle8192
gemma-2-27b-instruct27.2 billionGemmaGoogle8192
qwen2-7b7.62 billionQwenTongyi Qianwen131072
qwen2-7b-instruct7.62 billionQwenTongyi Qianwen131072
qwen2-72b72.7 billionQwenTongyi Qianwen131072
qwen2-72b-instruct72.7 billionQwenTongyi Qianwen131072

*Gemma-7b models are not available to deploy in the Developer tier since an A10G is not able to support its requirements effectively.

Best-effort LLMs

Predibase provides best-effort support for any Huggingface LLM meeting the following criteria:

How to Deploy a Custom LLM

  1. Get the Huggingface ID for your model by clicking the the copy icon on the custom base model's page, ex. "BioMistral/BioMistral-7B".

Huggingface screenshot

  1. Pass the Huggingface ID as the base_model, the appropriate accelerator ID for accelerator based on your tier or contract, and hf_token (your Huggingface token) if deploying a private model.
pb.deployments.create(
name="my-biomistral-7b",
config=DeploymentConfig(
base_model="BioMistral/BioMistral-7B",
accelerator="a10_24gb_100", # Required for custom models
# hf_token="<YOUR HUGGINGFACE TOKEN>" # Required for private Huggingface models
# cooldown_time=3600, # Value in seconds, defaults to 43200 (12hrs), set to 0 for always-on.
min_replicas=0, # Auto-scales to 0 replicas when not in use
max_replicas=1
)
)
  1. Prompt your adapter as normal.

Instruction Templates

The following instruction templates are used in the UI when prompting our shared serverless deployments. When using the SDK or REST API for inference, you will need to include these templates yourself in the prompt, otherwise you may see less than stellar responses.

Llama 3 models

Instruct models

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a helpful, detailed, and polite artificial intelligence assistant. Your answers are clear and suitable for a professional environment.

If context is provided, answer using only the provided contextual information.<|eot_id|><|start_header_id|>user<|end_header_id|>

<insert your prompt here><|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n

Non-instruct models

None

Llama 2 models

Chat models

<<SYS>>
You are a helpful, detailed, and polite artificial intelligence assistant. Your answers are clear and suitable for a professional environment.

If context is provided, answer using only the provided contextual information.
<</SYS>>

[INST] <insert your prompt here> [/INST]

Non-chat models

None

Codellama models

codellama-13b-instruct

<s>[INST] <insert your prompt here> [/INST]

codellama-70b-instruct

<s>Source: user\n\n <insert your prompt here> <step> Source: assistant\nDestination: user\n\n

Mistral & Mixtral models

<<SYS>>
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.

If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
<</SYS>>

[INST] <insert your prompt here> [/INST]

Solar models

Instruct models

<|im_start|>user\n <insert your prompt here> <|im_end|>\n<|im_start|>assistant\n

Non-instruct models

None.

Gemma models

Instruct models

<start_of_turn>user
<insert your prompt here><end_of_turn>
<start_of_turn>model

Non-instruct models

None

Phi-2

<|im_start|>user\n<insert your prompt here><|im_end|>\n

Zephyr-7b-beta

<|system|>
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.

If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.</s>
<|user|>
<insert your prompt here></s>
<|assistant|>