Chat Templates
Overview
Open source models typically come in two versions:
- a pre-trained base model (e.g., Llama-3-8B) and
- an instruct version (e.g., Llama-3-8B-Instruct).
The instruct version undergoes further training with specific instructions using a chat template. These templates ensure clarity and consistency in instructions, leading to better fine-tuning outcomes. They also focus the model's learning on relevant aspects of the data.
Experimentation shows that using model-specific chat templates significantly boosts performance. While custom templates can be used with base models, model-specific instruction formatting usually results in better performance for instruction tuning both on the base model, but especially on instruction-tuned base models.
Note: Instruction tuning templates are very sensitive to spacing and new line characters. It is also important to use the same instruction tuning formatting from training during inference, otherwise you are very likely to see poor results.
See how to format your dataset for different base models below. If you don't see a model that we support in the base models below, then an instruct template may not be required.
[NEW] Automatically Apply the Chat Template For Finetuning
For fine-tuning, apply_chat_template
is now supported in the FinetuningConfig:
config=FinetuningConfig(
base_model="mistral-7b",
adapter="turbo_lora", # default: "lora"; Turbo LoRA is a proprietary fine-tuning method which greatly improves inference throughput for longer output token tasks.
epochs=1, # default: 3
rank=8, # default: 16
learning_rate=0.0001, # default: 0.0002
target_modules=["q_proj", "v_proj", "k_proj"], # default: None (infers [q_proj, v_proj] for mistral-7b)
apply_chat_template=True, # default: False
),
When this parameter is set to True
, each training sample in the dataset will automatically have the model's chat template applied to it. Note that this parameter is only supported for instruction and chat fine-tuning, not text completion.
Inference with a Chat Template
If your model was trained with apply_chat_template
set to True
, please use only the OpenAI chat completions API to query the model because the chat template will automatically be applied to your inputs:
from openai import OpenAI
api_token = "<PREDIBASE API TOKEN>"
tenant_id = "<PREDIBASE TENANT ID>"
model_name = "<DEPLOYMENT NAME>" # Ex. "mistral-7b"
adapter = "<ADAPTER REPO NAME>/<VERSION NUMBER>" # Ex. "adapter-repo/1"
base_url = f"https://serving.app.predibase.com/{tenant_id}/deployments/v2/llms/{model_name}/v1"
client = OpenAI(
api_key=api_token,
base_url=base_url,
)
content = "<YOUR PROMPT>"
completion = client.chat.completions.create(
model=adapter,
messages=[{"role": "user", "content": content}],
max_tokens=100,
)
print("Completion result:", completion.choices[0].text)
Chat Templates for Base Models
- Llama 2
- Llama 3
- Mistral & Mixtral
- Zephyr
- Solar
- Gemma
- CodeLlama
- Phi-3
- Phi-3.5
- Qwen2
- Gemma-2-Instruct
Models
- llama-2-7b-chat
- llama-2-13b-chat
- llama-2-70b-chat
Input Prompt Template
There are two options, depending on whether you want to also include a system message as part of the instruction.
With system message:
<s>[INST] <<SYS>>\n {system_message} \n<</SYS>>\n\n {prompt} [/INST]
Without system message:
<s>[INST] {prompt} [/INST]
Make sure there is a space after the final [/INST] tag.
Output Prompt Template
You don't have to modify the completion column.
Example Data
Raw data without the template:
Prompt | Completion |
---|---|
Please answer the following question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? | Natalia sold 48/2 = <<48/2=24>>24 clips in May. Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. Answer: 72 |
Transformed data after applying the chat template without a system message:
Prompt | Completion |
---|---|
<s>[INST] Please answer the following question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? [/INST] | Natalia sold 48/2 = <<48/2=24>>24 clips in May. Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. Answer: 72 |
Models
- llama-3-8b-instruct
- llama-3-70b-instruct
- llama-3-1-8b-instruct
Input Prompt Template
There are two options, depending on whether you want to also include a system message as part of the instruction.
With system message:
<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n {system_message} <|eot_id|><|start_header_id|>user<|end_header_id|>\n\n {prompt} <|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n
Without system message:
<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\n {prompt} <|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n
Make sure there are two \n
characters after the final <|end_header_id>
tag.
Output Prompt Template
You don't have to modify the completion column.
Example Data
Raw data without the template:
Prompt | Completion |
---|---|
Please answer the following question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? | Natalia sold 48/2 = <<48/2=24>>24 clips in May. Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. Answer: 72 |
Transformed data after applying the chat template without a system message:
Prompt:
<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\n Please answer the following question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? <|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n
Completion:
Natalia sold 48/2 = <<48/2=24>>24 clips in May. Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. Answer: 72
Models
- mistral-7b-instruct
- mistral-7b-instruct-v0-2
- mistral-7b-instruct-v0-3
- mistral-nemo-12b-instruct-2407
- mixtral-8x7b-instruct-v0-1
Input Prompt Template
These models don't support using a system message.
<s>[INST] {prompt} [/INST]
Make sure there is a space after the final [/INST] tag.
Output Prompt Template
You don't have to modify the completion column.
Example Data
Raw data without the template:
Prompt | Completion |
---|---|
Please answer the following question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? | Natalia sold 48/2 = <<48/2=24>>24 clips in May. Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. Answer: 72 |
Transformed data after applying the chat template without a system message:
Prompt | Completion |
---|---|
<s>[INST] Please answer the following question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? [/INST] | Natalia sold 48/2 = <<48/2=24>>24 clips in May. Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. Answer: 72 |
Models
- zephyr-7b-beta
Input Prompt Template
There are two options, depending on whether you want to also include a system message as part of the instruction.
With system message:
<|system|>\n {system_message} </s>\n<|user|>\n {prompt} </s>\n<|assistant|>\n
Without system message:
<|user|>\n {prompt} </s>\n<|assistant|>\n
Make sure there is a \n
after the final <|assistant|>
tag.
Output Prompt Template
You don't have to modify the completion column.
Example Data
Raw data without the template:
Prompt | Completion |
---|---|
Please answer the following question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? | Natalia sold 48/2 = <<48/2=24>>24 clips in May. Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. Answer: 72 |
Transformed data after applying the chat template without a system message:
Prompt:
<|user|>\n Please answer the following question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? </s>\n<|assistant|>\n
Completion:
Natalia sold 48/2 = <<48/2=24>>24 clips in May. Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. Answer: 72
Models
- solar-1-mini-chat-240612
- solar-pro-preview-instruct
Input Prompt Template
There are two options, depending on whether you want to also include a system message as part of the instruction.
With system message:
<|im_start|>system\n {system_message} <|im_end|>\n<|im_start|>user\n {prompt} <|im_end|>\n<|im_start|>assistant\n
Without system message:
<|im_start|>user\n {prompt} <|im_end|>\n<|im_start|>assistant\n
Make sure there is a \n
after assistant at the end of the template.
Output Prompt Template
You don't have to modify the completion column.
Example Data
Raw data without the template:
Prompt | Completion |
---|---|
Please answer the following question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? | Natalia sold 48/2 = <<48/2=24>>24 clips in May. Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. Answer: 72 |
Transformed data after applying the chat template without a system message:
Prompt:
<|im_start|>user\n Please answer the following question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? <|im_end|>\n<|im_start|>assistant\n
Completion:
Natalia sold 48/2 = <<48/2=24>>24 clips in May. Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. Answer: 72
Models
- gemma-2b-it
- gemma-7b-it
Input Prompt Template
These models don't support using a system message.
<bos><start_of_turn>user\n {prompt} <end_of_turn>\n<start_of_turn>model\n
Make sure there is a \n
character after model
.
Output Prompt Template
You don't have to modify the completion column.
Example Data
Raw data without the template:
Prompt | Completion |
---|---|
Please answer the following question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? | Natalia sold 48/2 = <<48/2=24>>24 clips in May. Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. Answer: 72 |
Transformed data after applying the chat template without a system message:
Prompt | Completion |
---|---|
<bos><start_of_turn>user\n Please answer the following question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? <end_of_turn>\n<start_of_turn>model\n | Natalia sold 48/2 = <<48/2=24>>24 clips in May. Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. Answer: 72 |
Models
- codellama-7b-instruct
- codellama-13b-instruct
- codellama-70b-instruct
Input Prompt Template
There are two options, depending on whether you want to also include a system message as part of the instruction.
With system message:
<s>[INST] <<SYS>>\n {system_message} \n<</SYS>>\n\n {prompt} [/INST]
Without system message:
<s>[INST] {prompt} [/INST]
Make sure there is a space after the final [/INST] tag.
Output Prompt Template
You don't have to modify the completion column.
Example Data
Raw data without the template:
Prompt | Completion |
---|---|
Please answer the following question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? | Natalia sold 48/2 = <<48/2=24>>24 clips in May. Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. Answer: 72 |
Transformed data after applying the chat template without a system message:
Prompt | Completion |
---|---|
<s>[INST] Please answer the following question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? [/INST] | Natalia sold 48/2 = <<48/2=24>>24 clips in May. Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. Answer: 72 |
Models
- phi-3-mini-4k-instruct
Input Prompt Template
This model does not support using a system message.
<|user|>\n {prompt} <|end|>\n<|assistant|>\n
Make sure there is a \n
after assistant at the end of the template.
Output Prompt Template
You don't have to modify the completion column.
Example Data
Raw data without the template:
Prompt | Completion |
---|---|
Please answer the following question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? | Natalia sold 48/2 = <<48/2=24>>24 clips in May. Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. Answer: 72 |
Transformed data after applying the chat template:
Prompt:
<s><|user|>\n Please answer the following question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? <|end|>\n<|assistant|>\n
Completion:
Natalia sold 48/2 = <<48/2=24>>24 clips in May. Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. Answer: 72
Models
- phi-3-5-mini-instruct
Input Prompt Template
There are two options, depending on whether you want to also include a system message as part of the instruction.
With system message:
<|system|>\n {system_message} <|end|>\n<|user|>\n {prompt} <|end|>\n<|assistant|>\n
Without system message:
<|user|>\n {prompt} <|end|>\n<|assistant|>\n
:::
Output Prompt Template
You don't have to modify the completion column.
Example Data
Raw data without the template:
Prompt | Completion |
---|---|
Please answer the following question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? | Natalia sold 48/2 = <<48/2=24>>24 clips in May. Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. Answer: 72 |
Transformed data after applying the chat template without a system message:
Prompt:
<|user|>\n Please answer the following question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? <|end|>\n<|assistant|>\n
Completion:
Natalia sold 48/2 = <<48/2=24>>24 clips in May. Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. Answer: 72
Models
- qwen2-1-5b
- qwen2-1-5b-instruct
- qwen2-7b
- qwen2-7b-instruct
- qwen2-72b
- qwen2-72b-instruct
Input Prompt Template
There are two options, depending on whether you want to also include a system message as part of the instruction.
With system message:
<|im_start|>system\nThis is a system message<|im_end|>\n<|im_start|>user\nThis is a user message<|im_end|>\n<|im_start|>assistant\n
Without system message:
<|im_start|>user\nThis is a user message<|im_end|>\n<|im_start|>assistant\n
Make sure there is a \n after the "assistant" tag
Output Prompt Template
You don't have to modify the completion column.
Example Data
Raw data without the template:
Prompt | Completion |
---|---|
Please answer the following question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? | Natalia sold 48/2 = <<48/2=24>>24 clips in May. Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. Answer: 72 |
Transformed data after applying the chat template without a system message:
Prompt | Completion |
---|---|
<|im_start|>user\n Please answer the following question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? <|im_end|>\n<|im_start|>assistant\n | Natalia sold 48/2 = <<48/2=24>>24 clips in May. Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. Answer: 72 |
Models
- gemma-2-9b-instruct
- gemma-2-27b-instruct
Input Prompt Template
<bos><start_of_turn>user\n{prompt}<end_of_turn>\n<start_of_turn>model\n
Output Prompt Template
You don't have to modify the completion column.
Example Data
Raw data without the template:
Prompt | Completion |
---|---|
Please answer the following question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? | Natalia sold 48/2 = <<48/2=24>>24 clips in May. Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. Answer: 72 |
Transformed data after applying the chat template:
Prompt | Completion |
---|---|
\<bos><start_of_turn>user\nPlease answer the following question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?<end_of_turn>\n<start_of_turn>model\n | Natalia sold 48/2 = <<48/2=24>>24 clips in May. Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May. Answer: 72 |