Skip to main content

pb.deployments.create

Note: This method is for creating a private serverless deployment. You can also query base or fine-tuned models via shared endpoints.

pb.deployments.create

     Create a new private serverless deployment

Parameters:​

       name: str
       Name of the private serverless deployment

       description: str, default None
       Description for the deployment

       config: Deployment Config

Returns:​

       Deployment

Example:​

Create a new private serverless deployment

pb.deployments.create(
    name="my-mistral-7b",
    config=DeploymentConfig(
        base_model="mistral-7b-instruct-v0-2",
        # cooldown_time=3600,  # Value in seconds, defaults to 3600 (1hr)
        min_replicas=0,  # Auto-scales to 0 replicas when not in use
        max_replicas=1
    )
    # description="", # Optional
)

Using a few different configuration parameters:

pb.deployments.create(
  name="my-llama-3-8b",
  config=DeploymentConfig(
    base_model="llama-3-8b",
    max_total_tokens=4094,  # Change the default context window size
    quantization="fp8",  # Enable quantization
    requests_logging_enabled=True,  # Enable request logging
    preloaded_adapters=["my-adapter/1", "my-adapter/2"],  # Preload adapters for performance
    prefix_caching=True  # Enable prefix caching
  )
)

Notes