Example: Deploy to Azure ML

Make sure you have the Predibase SDK installed. Connect to your Predibase instance with the API token generated on the Settings page:

from getpass import getpass
from predibase import PredibaseClient

pc = PredibaseClient(token=getpass("PREDIBASE_API_TOKEN: "))

Download from Predibase

Download the model that you want to deploy from Predibase. Specify the framework to use for the deployment. In this case, we are using the Triton Inference Server framework from NVIDIA. Also, specify the location on your local machine where you want to save the model.

model = pc.get_model("QQA: Ethos Binary")

MODEL_LOCATION = "/tmp"
model.download(f"{model.id}.tar", location=MODEL_LOCATION, framework="triton")

Extract the model

import os
import tarfile

with tarfile.open(os.path.join(MODEL_LOCATION, f"{model.id}.tar"), "r") as tar:
    tar.extractall(os.path.join(MODEL_LOCATION, f"{model.id}"))

Make the model name URL friendly

import re
import unicodedata


def slugify(value: str) -> str:
    value = unicodedata.normalize("NFKC", value)
    value = re.sub(r"[^\w\s-]", "", value).strip().lower()
    return re.sub(r"[-\s]+", "-", value)


def replace_in_file(path: str, old: str, new: str):
    with open(path, "r") as f:
        config = f.read()
    config = config.replace(old, new)
    with open(path, "w") as f:
        f.write(config)


def slugify_dir(path: str):
    """Recursively slugify directory names in a given path."""
    for root, dirs, files in os.walk(path):
        for d in dirs:
            new_d = slugify(d)

            if new_d == d:
                continue
            print(f"Renaming '{d}' to '{new_d}'")

            os.rename(os.path.join(root, d), os.path.join(root, new_d))
            slugify_dir(os.path.join(root, new_d))

            # Replace the directory name in the config.pbtxt file
            config_path = os.path.join(root, new_d, "config.pbtxt")
            if os.path.exists(config_path):
                replace_in_file(config_path, d, new_d)

slugify_dir(os.path.join(MODEL_LOCATION, f"{model.id}"))

Renaming 'QQA: Ethos Binary' to 'qqa-ethos-binary'
Renaming 'QQA: Ethos Binary_predictor' to 'qqa-ethos-binary_predictor'
Renaming 'QQA: Ethos Binary_preprocessor' to 'qqa-ethos-binary_preprocessor'
Renaming 'QQA: Ethos Binary_postprocessor' to 'qqa-ethos-binary_postprocessor'

Deploy to Azure ML

Now that we have the model, we can deploy it to Azure ML. You will need to have an Azure ML workspace set up.

This part of the notebook is based on the Azure ML documentation for deploying Triton models.

pip install azure-cli azure-ai-ml

az login --use-device-code

Enter the subscription ID, resource group, and workspace name for your Azure ML workspace:

subscription_id = ""
resource_group = ""
workspace_name = ""

from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential

ml_client = MLClient(DefaultAzureCredential(), subscription_id, resource_group, workspace_name)

Create the REST endpoint for your model:

from azure.ai.ml.entities import ManagedOnlineEndpoint

ENDPOINT_NAME = "predibase"
endpoint = ManagedOnlineEndpoint(name=ENDPOINT_NAME, auth_mode="key")

ml_client.online_endpoints.begin_create_or_update(endpoint)

Deploy your model to the REST endpoint:

from azure.ai.ml.entities import ManagedOnlineDeployment, Model
from azure.ai.ml.constants import AssetTypes

DEPLOYMENT_NAME = slugify(model.repo.name) + "-" + str(model.version)
deployment = ManagedOnlineDeployment(
    name=DEPLOYMENT_NAME,
    endpoint_name=ENDPOINT_NAME,
    model=Model(
        name=slugify(model.repo.name),
        version=str(model.version),
        path=MODEL_LOCATION + str(model.id),
        type=AssetTypes.TRITON_MODEL,
    ),
    instance_type="Standard_F2s_v2",
    instance_count=1,
)

ml_client.online_deployments.begin_create_or_update(deployment)

Assign all traffic to the new deployment

endpoint.traffic = {DEPLOYMENT_NAME: 100}
ml_client.online_endpoints.begin_create_or_update(endpoint)

Predict

Now that we have deployed the model, we can use it to make predictions.

Using Predibase SDK

We can use the Predibase SDK to conveniently make predictions on a Pandas dataframe.

First, we need to install the additional predictor dependencies for the SDK:

pip install -U "predibase[predictor]"

Now we can make predictions on a sample Pandas dataframe:

import pandas as pd

df = pd.DataFrame({"comment": ["I love this product", "I hate this product"]})

	id	comment
0	1	I love this product
1	2	I hate this product

keys = ml_client.online_endpoints.get_keys(ENDPOINT_NAME)
AZURE_API_KEY = keys.primary_key

from predibase import Predictor
from predibase.pql import Session

predictor = Predictor(Session(token=AZURE_API_KEY), slugify(model.repo.name))
predictor.http_endpoint = f"{ENDPOINT_NAME}.eastus2.inference.ml.azure.com"

preds = predictor.predict(df.head())
preds.to_pandas()

	isHate_predictions	isHate_probabilities
0	False	[0.9842461, 0.015753938]
1	True	[2.3841858e-07, 0.99999976]

Using Python standard library

Alternatively, we can use the Python standard library to make predictions.

import requests

data = {
    "inputs": [
        {
            "name": "comment",
            "shape": [1],
            "datatype": "BYTES",
            "data": ["I love this product"],
        }
    ]
}
headers = {"Authorization": f"Bearer {AZURE_API_KEY}"}
url = f"https://{ENDPOINT_NAME}.eastus2.inference.ml.azure.com/v2/models/{slugify(model.repo.name)}/infer"

response = requests.post(url, json=data, headers=headers)

Clean up

When you're done, remove the endpoint to avoid unexpected charges to your Azure account:

ml_client.online_endpoints.begin_delete(name=ENDPOINT_NAME)