Fine-tuning Models
Well-supported LLMs
Fine-tuning is currently available for the following models:
| Model Name | Parameters | Architecture | License | Context Window | Supported Fine-Tuning Context Window |
|---|---|---|---|---|---|
| mistral-7b | 7 billion | Mistral | Apache 2.0 | 32768 | 32768 |
| mistral-7b-instruct | 7 billion | Mistral | Apache 2.0 | 32768 | 32768 |
| mistral-7b-instruct-v0-2 | 7 billion | Mistral | Apache 2.0 | 32768 | 32768 |
| mixtral-8x7b | 46.7 billion | Mixtral | Apache 2.0 | 32768 | 7168 |
| mixtral-8x7b-instruct-v0-1 | 46.7 billion | Mixtral | Apache 2.0 | 32768 | 7168 |
| llama-3-8b | 8 billion | Llama-3 | Meta (request for commercial use) | 8192 | 8192 |
| llama-3-8b-instruct | 8 billion | Llama-3 | Meta (request for commercial use) | 8192 | 8192 |
| llama-3-70b | 70 billion | Llama-3 | Meta (request for commercial use) | 8192 | 8192 |
| llama-3-70b-instruct | 70 billion | Llama-3 | Meta (request for commercial use) | 8192 | 8192 |
| llama-2-7b | 7 billion | Llama-2 | Meta (request for commercial use) | 4096 | 4096 |
| llama-2-7b-chat | 7 billion | Llama-2 | Meta (request for commercial use) | 4096 | 4096 |
| llama-2-13b | 13 billion | Llama-2 | Meta (request for commercial use) | 4096 | 4096 |
| llama-2-13b-chat | 13 billion | Llama-2 | Meta (request for commercial use) | 4096 | 4096 |
| llama-2-70b | 70 billion | Llama-2 | Meta (request for commercial use) | 4096 | 4096 |
| llama-2-70b-chat | 70 billion | Llama-2 | Meta (request for commercial use) | 4096 | 4096 |
| codellama-13b-instruct | 13 billion | Llama-2 | Meta (request for commercial use) | 16384 | 16384 |
| codellama-70b-instruct | 70 billion | Llama-2 | Meta (request for commercial use) | 4096 | 4096 |
| gemma-2b | 2 billion | Gemma | 8192 | 8192 | |
| gemma-2b-instruct | 2 billion | Gemma | 8192 | 8192 | |
| gemma-7b | 7 billion | Gemma | 8192 | 8192 | |
| gemma-7b-instruct | 7 billion | Gemma | 8192 | 8192 | |
| zephyr-7b-beta | 7 billion | Mistral | MIT | 32768 | 32768 |
| phi-2 | 2.78 billion | Phi | Microsoft | 2048 | 2048 |
Many of the latest OSS models are released in two variants:
- Base model (llama-2-7b, etc): These are models that are primarily trained on the objective of text completion.
- Instruction-Tuned (llama-2-7b-chat, mistral-7b-instruct, etc): These are models that have been further trained on (instruction, output) pairs in order to better respond to human instruction-styled inputs. The instructions effectively constrains the model’s output to align with the response characteristics or domain knowledge.
Best-Effort LLMs (via HuggingFace)
Fine-tuning is also supported for any Huggingface LLM meeting the following criteria:
- Has the "Text Generation" and "Transformer" tags
- Does not have a "custom_code" tag
- Are not post-quantized (ex. model containing a quantization method such as "AWQ" in the name)
- Has text inputs and outputs
Note: Best-effort indicates we will try to support these models but it is not guaranteed.