After fine-tuning completes, you may choose to continue training on top of the fine-tuned adapter rather than starting from scratch. This is useful when you want to:

  • See if your adapter’s performance improves with further training on the same data
  • Fine-tune on a similar dataset without starting over
  • Resume training from a specific checkpoint

​
Starting a Continued Training Run

To continue training from an existing adapter, provide the adapter ID in your configuration:

from predibase import SFTConfig

# Continue from the latest checkpoint
adapter = pb.adapters.create(
    config=SFTConfig(
        epochs=3,  # The maximum number of ADDITIONAL epochs to train for
        enable_early_stopping=False,
    ),
    continue_from_version="myrepo/3",  # The adapter version to resume training from
    dataset="mydataset",
    repo="myrepo"
)

# Continue from a specific checkpoint
adapter = pb.adapters.create(
    config=SFTConfig(
        epochs=3,  # The maximum number of ADDITIONAL epochs to train for
        enable_early_stopping=False,
    ),
    continue_from_version="myrepo/3@11",  # Resumes from checkpoint 11 of `myrepo/3`
    dataset="mydataset",
    repo="myrepo"
)

​
Configuration Options

When continuing training, only two parameters can be modified:

  • epochs or train_steps: Number of additional epochs or training steps to run. Only one of these parameters can be modified.
  • enable_early_stopping: Whether to enable early stopping.

All other parameters and hyperparameters are inherited from the original training run to ensure consistency.

​
Training Progress

​
Training on the Same Dataset

When continuing training on the same dataset:

  • Training progress from the previous run is preserved
  • Checkpoints and metrics are maintained
  • Training picks up exactly where it left off
  • Optimizer, learning rate scheduler, and RNG state are restored

​
Training on a Different Dataset

You can also use an existing adapter as the starting point for fine-tuning on a new dataset:

from predibase import SFTConfig

adapter = pb.adapters.create(
    config=SFTConfig(
        epochs=3,  # The maximum number of epochs to train for
        enable_early_stopping=False,
    ),
    continue_from_version="myrepo/3",  # The adapter version to resume training from
    dataset="mydataset-2",  # New dataset
    repo="myrepo"
)

When training on a new dataset:

  • Training starts fresh but uses the LoRA weights from the final checkpoint of the base run as initialization
  • Optimizer, learning rate scheduler, and RNG are re-initialized
  • Only epochs/train_steps and enable_early_stopping are configurable
  • Training progress from the base run is not preserved
  • Small warmup ratio is used to prevent catastrophic forgetting in the early stages of training

​
Next Steps