Continue Training

After fine-tuning completes, you may choose to continue training on top of the fine-tuned adapter rather than starting from scratch. This is useful when you want to:

• See if your adapter’s performance improves with further training on the same data
• Fine-tune on a similar dataset without starting over
• Resume training from a specific checkpoint

​

Starting a Continued Training Run

To continue training from an existing adapter, provide the adapter ID in your configuration:

from predibase import SFTConfig

# Continue from the latest checkpoint
adapter = pb.adapters.create(
    config=SFTConfig(
        epochs=3,  # The maximum number of ADDITIONAL epochs to train for
        enable_early_stopping=False,
    ),
    continue_from_version="myrepo/3",  # The adapter version to resume training from
    dataset="mydataset",
    repo="myrepo"
)

# Continue from a specific checkpoint
adapter = pb.adapters.create(
    config=SFTConfig(
        epochs=3,  # The maximum number of ADDITIONAL epochs to train for
        enable_early_stopping=False,
    ),
    continue_from_version="myrepo/3@11",  # Resumes from checkpoint 11 of `myrepo/3`
    dataset="mydataset",
    repo="myrepo"
)

​

Configuration Options

When continuing training, only two parameters can be modified:

• epochs or train_steps: Number of additional epochs or training steps to run. Only one of these parameters can be modified.
• enable_early_stopping: Whether to enable early stopping.

All other parameters and hyperparameters are inherited from the original training run to ensure consistency.

​

Training Progress

​

Training on the Same Dataset

When continuing training on the same dataset:

• Training progress from the previous run is preserved
• Checkpoints and metrics are maintained
• Training picks up exactly where it left off
• Optimizer, learning rate scheduler, and RNG state are restored

​

Training on a Different Dataset

You can also use an existing adapter as the starting point for fine-tuning on a new dataset:

from predibase import SFTConfig

adapter = pb.adapters.create(
    config=SFTConfig(
        epochs=3,  # The maximum number of epochs to train for
        enable_early_stopping=False,
    ),
    continue_from_version="myrepo/3",  # The adapter version to resume training from
    dataset="mydataset-2",  # New dataset
    repo="myrepo"
)

When training on a new dataset:

• Training starts fresh but uses the LoRA weights from the final checkpoint of the base run as initialization
• Optimizer, learning rate scheduler, and RNG are re-initialized
• Only epochs/train_steps and enable_early_stopping are configurable
• Training progress from the base run is not preserved
• Small warmup ratio is used to prevent catastrophic forgetting in the early stages of training

​

Next Steps

• Explore different adapter types
• Start evaluating your models