The ContinuedPretrainingConfig class provides configuration options for continued pretraining of language models using adapter-based approaches. This configuration class is essential when creating new adapters as it defines all the training parameters and hyperparameters needed for the continued pretraining process.

​
General Hyperparameters

ParameterTypeRequiredDefaultDescription
base_modelstringYesThe base model to fine-tune
adapterstringNoloraThe type of adapter to use.
One of lora, turbo_lora, turbo
epochsintegerNo3Number of epochs to train for
train_stepsintegerNoNumber of training steps (overrides epochs if set)
learning_ratefloatNo0.0002Learning rate for training
enable_early_stoppingbooleanNotrueWhether to enable early stopping
lr_schedulerobjectNocosine_with_restartsLearning rate scheduler configuration
optimizerobjectNoadamw_torchOptimizer configuration
warmup_ratiofloatNo0.03Ratio of training steps to use for warmup
effective_batch_sizeintegerNo16Effective batch size for training
apply_chat_templatebooleanNofalseWhether to apply the chat template

​
LoRA Specific Hyperparameters

ParameterTypeRequiredDefaultDescription
rankintegerNo16Rank of the LoRA adapter
target_modulesarray[string]NoTarget modules in attention blocksList of model modules to fine-tune
lora_alphaintegerNo2 x rankAlpha parameter for LoRA
lora_dropoutfloatNo0.00Dropout rate for LoRA

​
Example Usage

from predibase import ContinuedPretrainingConfig

# Basic configuration
config = ContinuedPretrainingConfig(
    base_model="qwen3-8b"
)

# Advanced configuration
config = ContinuedPretrainingConfig(
    base_model="qwen3-8b",
    task="instruction_tuning",
    epochs=1,
    rank=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"]
)