Example: Deploy to Azure ML
Make sure you have the Predibase SDK installed. Connect to your Predibase instance with the API token generated on the Settings page:
from getpass import getpass
from predibase import PredibaseClient
pc = PredibaseClient(token=getpass("PREDIBASE_API_TOKEN: "))
Download from Predibase
Download the model that you want to deploy from Predibase. Specify the framework to use for the deployment. In this case, we are using the Triton Inference Server framework from NVIDIA. Also, specify the location on your local machine where you want to save the model.
model = pc.get_model("QQA: Ethos Binary")
MODEL_LOCATION = "/tmp"
model.download(f"{model.id}.tar", location=MODEL_LOCATION, framework="triton")
Extract the model
import os
import tarfile
with tarfile.open(os.path.join(MODEL_LOCATION, f"{model.id}.tar"), "r") as tar:
tar.extractall(os.path.join(MODEL_LOCATION, f"{model.id}"))
Make the model name URL friendly
import re
import unicodedata
def slugify(value: str) -> str:
value = unicodedata.normalize("NFKC", value)
value = re.sub(r"[^\w\s-]", "", value).strip().lower()
return re.sub(r"[-\s]+", "-", value)
def replace_in_file(path: str, old: str, new: str):
with open(path, "r") as f:
config = f.read()
config = config.replace(old, new)
with open(path, "w") as f:
f.write(config)
def slugify_dir(path: str):
"""Recursively slugify directory names in a given path."""
for root, dirs, files in os.walk(path):
for d in dirs:
new_d = slugify(d)
if new_d == d:
continue
print(f"Renaming '{d}' to '{new_d}'")
os.rename(os.path.join(root, d), os.path.join(root, new_d))
slugify_dir(os.path.join(root, new_d))
# Replace the directory name in the config.pbtxt file
config_path = os.path.join(root, new_d, "config.pbtxt")
if os.path.exists(config_path):
replace_in_file(config_path, d, new_d)
slugify_dir(os.path.join(MODEL_LOCATION, f"{model.id}"))
Renaming 'QQA: Ethos Binary' to 'qqa-ethos-binary'
Renaming 'QQA: Ethos Binary_predictor' to 'qqa-ethos-binary_predictor'
Renaming 'QQA: Ethos Binary_preprocessor' to 'qqa-ethos-binary_preprocessor'
Renaming 'QQA: Ethos Binary_postprocessor' to 'qqa-ethos-binary_postprocessor'
Deploy to Azure ML
Now that we have the model, we can deploy it to Azure ML. You will need to have an Azure ML workspace set up.
This part of the notebook is based on the Azure ML documentation for deploying Triton models.
pip install azure-cli azure-ai-ml
az login --use-device-code
Enter the subscription ID, resource group, and workspace name for your Azure ML workspace:
subscription_id = ""
resource_group = ""
workspace_name = ""
from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential
ml_client = MLClient(DefaultAzureCredential(), subscription_id, resource_group, workspace_name)
Create the REST endpoint for your model:
from azure.ai.ml.entities import ManagedOnlineEndpoint
ENDPOINT_NAME = "predibase"
endpoint = ManagedOnlineEndpoint(name=ENDPOINT_NAME, auth_mode="key")
ml_client.online_endpoints.begin_create_or_update(endpoint)
Deploy your model to the REST endpoint:
from azure.ai.ml.entities import ManagedOnlineDeployment, Model
from azure.ai.ml.constants import AssetTypes
DEPLOYMENT_NAME = slugify(model.repo.name) + "-" + str(model.version)
deployment = ManagedOnlineDeployment(
name=DEPLOYMENT_NAME,
endpoint_name=ENDPOINT_NAME,
model=Model(
name=slugify(model.repo.name),
version=str(model.version),
path=MODEL_LOCATION + str(model.id),
type=AssetTypes.TRITON_MODEL,
),
instance_type="Standard_F2s_v2",
instance_count=1,
)
ml_client.online_deployments.begin_create_or_update(deployment)
Assign all traffic to the new deployment
endpoint.traffic = {DEPLOYMENT_NAME: 100}
ml_client.online_endpoints.begin_create_or_update(endpoint)
Predict
Now that we have deployed the model, we can use it to make predictions.
Using Predibase SDK
We can use the Predibase SDK to conveniently make predictions on a Pandas dataframe.
First, we need to install the additional predictor dependencies for the SDK:
pip install -U "predibase[predictor]"
Now we can make predictions on a sample Pandas dataframe:
import pandas as pd
df = pd.DataFrame({"comment": ["I love this product", "I hate this product"]})
id | comment | |
---|---|---|
0 | 1 | I love this product |
1 | 2 | I hate this product |
keys = ml_client.online_endpoints.get_keys(ENDPOINT_NAME)
AZURE_API_KEY = keys.primary_key
from predibase import Predictor
from predibase.pql import Session
predictor = Predictor(Session(token=AZURE_API_KEY), slugify(model.repo.name))
predictor.http_endpoint = f"{ENDPOINT_NAME}.eastus2.inference.ml.azure.com"
preds = predictor.predict(df.head())
preds.to_pandas()
isHate_predictions | isHate_probabilities | |
---|---|---|
0 | False | [0.9842461, 0.015753938] |
1 | True | [2.3841858e-07, 0.99999976] |
Using Python standard library
Alternatively, we can use the Python standard library to make predictions.
import requests
data = {
"inputs": [
{
"name": "comment",
"shape": [1],
"datatype": "BYTES",
"data": ["I love this product"],
}
]
}
headers = {"Authorization": f"Bearer {AZURE_API_KEY}"}
url = f"https://{ENDPOINT_NAME}.eastus2.inference.ml.azure.com/v2/models/{slugify(model.repo.name)}/infer"
response = requests.post(url, json=data, headers=headers)
Clean up
When you're done, remove the endpoint to avoid unexpected charges to your Azure account:
ml_client.online_endpoints.begin_delete(name=ENDPOINT_NAME)