Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.predibase.com/llms.txt

Use this file to discover all available pages before exploring further.

​
Well-supported LLMs

The following models are currently well-supported for fine-tuning.

​
Large Language Models (Text)

You may fine-tune LoRA, Turbo LoRA, and Turbo adapters on any of these base LLMs, but note that for Turbo LoRA and Turbo adapters, some models may require additional deployment configurations.

​
Qwen Models

Model NameParametersContext WindowSupported Fine-Tuning Context WindowAdapter Pre-load Not RequiredSupports GRPOArchitectureLicense
qwen3-30b-a3b30.5 billion13107232768❌✅QwenTongyi Qianwen
qwen3-32b32.8 billion13107232768❌✅QwenTongyi Qianwen
qwen3-14b14.8 billion13107232768❌✅QwenTongyi Qianwen
qwen3-8b8.19 billion13107232768❌✅QwenTongyi Qianwen
qwen2-5-32b32.8 billion13107216384❌✅QwenTongyi Qianwen
qwen2-5-32b-instruct32.8 billion3276816384❌✅QwenTongyi Qianwen
qwen2-5-14b14.8 billion13107232768❌✅QwenTongyi Qianwen
qwen2-5-14b-instruct14.8 billion3276832768❌✅QwenTongyi Qianwen
qwen2-5-7b7.62 billion13107232768❌✅QwenTongyi Qianwen
qwen2-5-7b-instruct7.62 billion3276832768❌✅QwenTongyi Qianwen
qwen2-5-3b3.09 billion13107232768❌✅QwenTongyi Qianwen
qwen2-5-3b-instruct3.09 billion3276832768❌✅QwenTongyi Qianwen
qwen2-5-1-5b1.54 billion13107232768❌✅QwenTongyi Qianwen
qwen2-5-1-5b-instruct1.54 billion3276832768❌✅QwenTongyi Qianwen
qwen2-5-coder-32b-instruct32.8 billion13107216384❌✅QwenTongyi Qianwen
qwen2-5-coder-7b-instruct7.62 billion13107216384❌✅QwenTongyi Qianwen
qwen2-5-coder-3b-instruct3.09 billion13107216384❌✅QwenTongyi Qianwen
qwen2-7b7.62 billion13107232768❌❌QwenTongyi Qianwen
qwen2-1-5b-instruct1.54 billion13107232768❌❌QwenTongyi Qianwen
qwen2-1-5b1.54 billion13107232768❌❌QwenTongyi Qianwen

​
Llama 3 Models

Model NameParametersContext WindowSupported Fine-Tuning Context WindowAdapter Pre-load Not RequiredSupports GRPOArchitectureLicense
llama-3-3-70b70.6 billion1310728192❌❌Llama-3Meta (request for commercial use)
llama-3-2-3b3.21 billion3276832768❌✅Llama-3Meta (request for commercial use)
llama-3-2-3b-instruct3.21 billion3276832768❌✅Llama-3Meta (request for commercial use)
llama-3-2-1b1.24 billion3276832768❌✅Llama-3Meta (request for commercial use)
llama-3-2-1b-instruct1.24 billion3276832768❌✅Llama-3Meta (request for commercial use)
llama-3-1-8b8 billion6299932768❌✅Llama-3Meta (request for commercial use)
llama-3-1-8b-instruct8 billion6299932768✅✅Llama-3Meta (request for commercial use)
llama-3-70b70 billion81928192❌❌Llama-3Meta (request for commercial use)
llama-3-70b-instruct70 billion81928192❌❌Llama-3Meta (request for commercial use)
llama-3-8b8 billion81928192❌❌Llama-3Meta (request for commercial use)
llama-3-8b-instruct8 billion81928192❌❌Llama-3Meta (request for commercial use)

​
Llama 2 Models

Model NameParametersContext WindowSupported Fine-Tuning Context WindowAdapter Pre-load Not RequiredSupports GRPOArchitectureLicense
llama-2-70b70 billion40964096❌❌Llama-2Meta (request for commercial use)
llama-2-70b-chat70 billion40964096❌❌Llama-2Meta (request for commercial use)
llama-2-13b13 billion40964096❌❌Llama-2Meta (request for commercial use)
llama-2-13b-chat13 billion40964096❌❌Llama-2Meta (request for commercial use)
llama-2-7b7 billion40964096❌❌Llama-2Meta (request for commercial use)
llama-2-7b-chat7 billion40964096❌❌Llama-2Meta (request for commercial use)

​
CodeLlama Models

Model NameParametersContext WindowSupported Fine-Tuning Context WindowAdapter Pre-load Not RequiredSupports GRPOArchitectureLicense
codellama-70b-instruct70 billion40964096❌❌Llama-2Meta (request for commercial use)
codellama-13b-instruct13 billion1638416384❌❌Llama-2Meta (request for commercial use)
codellama-7b7 billion40964096❌❌Llama-2Meta (request for commercial use)
codellama-7b-instruct7 billion40964096❌❌Llama-2Meta (request for commercial use)

​
Mistral Models

Model NameParametersContext WindowSupported Fine-Tuning Context WindowAdapter Pre-load Not RequiredSupports GRPOArchitectureLicense
mistral-7b-instruct-v0-37 billion3276832768✅❌MistralApache 2.0
mistral-7b-instruct-v0-27 billion3276832768✅❌MistralApache 2.0
mistral-7b7 billion3276832768✅❌MistralApache 2.0
mistral-7b-instruct7 billion3276832768✅❌MistralApache 2.0
mistral-nemo-12b-240712 billion13107232768❌❌MistralApache 2.0
mistral-nemo-12b-instruct-240712 billion13107232768❌❌MistralApache 2.0
zephyr-7b-beta7 billion3276832768✅❌MistralMIT
mixtral-8x7b-instruct-v0-146.7 billion327687168❌❌MixtralApache 2.0

​
Solar Models

Model NameParametersContext WindowSupported Fine-Tuning Context WindowAdapter Pre-load Not RequiredSupports GRPOArchitectureLicense
solar-1-mini-chat-24061210.7 billion3276832768✅❌LlamaCustom License
solar-pro-preview-instruct-v222.1 billion40964096❌❌SolarCustom License
solar-pro-24112622.1 billion3276816384❌❌SolarCustom License

​
Gemma Models

Model NameParametersContext WindowSupported Fine-Tuning Context WindowAdapter Pre-load Not RequiredSupports GRPOArchitectureLicense
gemma-2-27b27.2 billion81924096❌❌GemmaGoogle
gemma-2-27b-instruct27.2 billion81924096❌❌GemmaGoogle
gemma-2-9b9.24 billion81928192❌❌GemmaGoogle
gemma-2-9b-instruct9.24 billion81928192❌❌GemmaGoogle
gemma-7b8.5 billion81928192❌❌GemmaGoogle
gemma-7b-instruct8.5 billion81928192❌❌GemmaGoogle
gemma-2b2.5 billion81928192❌❌GemmaGoogle
gemma-2b-instruct2.5 billion81928192❌❌GemmaGoogle

​
Phi Models

Model NameParametersContext WindowSupported Fine-Tuning Context WindowAdapter Pre-load Not RequiredSupports GRPOArchitectureLicense
phi-3-5-mini-instruct3.8 billion13107216384❌✅Phi-3Microsoft
phi-3-mini-4k-instruct3.8 billion40964096Turbo LoRA not supported✅Phi-3Microsoft
phi-22.7 billion20482048Turbo not supported✅Phi-2Microsoft

​
Other Models

Model NameParametersContext WindowSupported Fine-Tuning Context WindowAdapter Pre-load Not RequiredSupports GRPOArchitectureLicense
deepseek-r1-distill-qwen-32b32.8 billion1310728000❌✅QwenDeepSeek-AI
openhands-lm-32b-v0.132.8 billion13107216384❌✅QwenXingyao Wang
Many of the latest OSS models are released in two variants:
  • Base model (llama-2-7b, etc): These are models that are primarily trained on the objective of text completion.
  • Instruction-Tuned (llama-2-7b-chat, mistral-7b-instruct, etc): These are models that have been further trained on (instruction, output) pairs in order to better respond to human instruction-styled inputs. The instructions effectively constrains the model’s output to align with the response characteristics or domain knowledge.

​
Vision Language Models

Currently, we support fine-tuning LoRAs for the following vision language models:

​
Qwen VL

Model NameParametersArchitectureLicenseLoRATurbo LoRATurboContext WindowSupported Fine-Tuning Context WindowSupports GRPO
qwen2-vl-7b-instruct7BQwen2Tongyi Qianwen✅❌❌12800032768❌
qwen2-5-vl-3b-instruct3BQwen2.5Tongyi Qianwen✅❌❌12800032768❌
qwen2-5-vl-7b-instruct7BQwen2.5Tongyi Qianwen✅❌❌12800032768❌
To get started with VLM fine-tuning, check out this user-guide here on how to format your data for training and test inference.

​
Best-Effort LLMs (via HuggingFace)

Best-effort support for fine-tuning is also offered for any Huggingface LLM meeting the following criteria:
  • Has the “Text Generation” and “Transformer” tags
  • Does not have a “custom_code” tag
  • Are not post-quantized (ex. model containing a quantization method such as “AWQ” in the name)
  • Has text inputs and outputs

​
Fine-tuning a custom LLM

  1. Get the Huggingface ID for your model by clicking the copy icon on the base model’s Huggingface page, ex. BioMistral/BioMistral-7B.
  2. Pass the Huggingface ID as the base_model.
from predibase import Predibase, SFTConfig
pb = Predibase(api_token=<API_TOKEN>)

# Create an adapter repository
repo = pb.repos.create(name="bio-summarizer", description="Bio News Summarizer", exists_ok=True)

# Start a fine-tuning job, blocks until training is finished
adapter = pb.adapters.create(
    config=SFTConfig(
        base_model="BioMistral/BioMistral-7B"
    ),
    dataset="bio-dataset",
    repo=repo,
    description="initial model with defaults"
)
Predibase training metrics will be automatically streamed to stdout. To view additional metrics via Tensorboard, pass show_tensorboard=True to the create call: 
from predibase import SFTConfig

adapter = pb.adapters.create(
    config=SFTConfig(
        base_model="BioMistral/BioMistral-7B"
    ),
    dataset="bio-dataset",
    repo=repo,
    description="initial model with defaults",
    show_tensorboard=True
)
Note that tensorboard data may take some time to refresh. Predibase also supports integrations with Weights & Biases and Comet.
Note that if you fine-tune a custom model not on our shared deployments list, you’ll need to deploy the custom base model as a private serverless deployment in order to run inference on your newly trained adapter. This is also supported on a best-effort basis.