Skip to main content

Example: Deploy to Azure ML

Make sure you have the Predibase SDK installed. Connect to your Predibase instance with the API token generated on the Settings page:

from getpass import getpass
from predibase import PredibaseClient

pc = PredibaseClient(token=getpass("PREDIBASE_API_TOKEN: "))

Download from Predibase

Download the model that you want to deploy from Predibase. Specify the framework to use for the deployment. In this case, we are using the Triton Inference Server framework from NVIDIA. Also, specify the location on your local machine where you want to save the model.

model = pc.get_model("QQA: Ethos Binary")

MODEL_LOCATION = "/tmp""{}.tar", location=MODEL_LOCATION, framework="triton")

Extract the model

import os
import tarfile

with, f"{}.tar"), "r") as tar:
tar.extractall(os.path.join(MODEL_LOCATION, f"{}"))

Make the model name URL friendly

import re
import unicodedata

def slugify(value: str) -> str:
value = unicodedata.normalize("NFKC", value)
value = re.sub(r"[^\w\s-]", "", value).strip().lower()
return re.sub(r"[-\s]+", "-", value)

def replace_in_file(path: str, old: str, new: str):
with open(path, "r") as f:
config =
config = config.replace(old, new)
with open(path, "w") as f:

def slugify_dir(path: str):
"""Recursively slugify directory names in a given path."""
for root, dirs, files in os.walk(path):
for d in dirs:
new_d = slugify(d)

if new_d == d:
print(f"Renaming '{d}' to '{new_d}'")

os.rename(os.path.join(root, d), os.path.join(root, new_d))
slugify_dir(os.path.join(root, new_d))

# Replace the directory name in the config.pbtxt file
config_path = os.path.join(root, new_d, "config.pbtxt")
if os.path.exists(config_path):
replace_in_file(config_path, d, new_d)
slugify_dir(os.path.join(MODEL_LOCATION, f"{}"))
Renaming 'QQA: Ethos Binary' to 'qqa-ethos-binary'
Renaming 'QQA: Ethos Binary_predictor' to 'qqa-ethos-binary_predictor'
Renaming 'QQA: Ethos Binary_preprocessor' to 'qqa-ethos-binary_preprocessor'
Renaming 'QQA: Ethos Binary_postprocessor' to 'qqa-ethos-binary_postprocessor'

Deploy to Azure ML

Now that we have the model, we can deploy it to Azure ML. You will need to have an Azure ML workspace set up.

This part of the notebook is based on the Azure ML documentation for deploying Triton models.

pip install azure-cli azure-ai-ml
az login --use-device-code

Enter the subscription ID, resource group, and workspace name for your Azure ML workspace:

subscription_id = ""
resource_group = ""
workspace_name = ""
from import MLClient
from azure.identity import DefaultAzureCredential

ml_client = MLClient(DefaultAzureCredential(), subscription_id, resource_group, workspace_name)

Create the REST endpoint for your model:

from import ManagedOnlineEndpoint

ENDPOINT_NAME = "predibase"
endpoint = ManagedOnlineEndpoint(name=ENDPOINT_NAME, auth_mode="key")

Deploy your model to the REST endpoint:

from import ManagedOnlineDeployment, Model
from import AssetTypes

DEPLOYMENT_NAME = slugify( + "-" + str(model.version)
deployment = ManagedOnlineDeployment(
path=MODEL_LOCATION + str(,

Assign all traffic to the new deployment

endpoint.traffic = {DEPLOYMENT_NAME: 100}


Now that we have deployed the model, we can use it to make predictions.

Using Predibase SDK

We can use the Predibase SDK to conveniently make predictions on a Pandas dataframe.

First, we need to install the additional predictor dependencies for the SDK:

pip install -U "predibase[predictor]"

Now we can make predictions on a sample Pandas dataframe:

import pandas as pd

df = pd.DataFrame({"comment": ["I love this product", "I hate this product"]})
01I love this product
12I hate this product
keys = ml_client.online_endpoints.get_keys(ENDPOINT_NAME)
AZURE_API_KEY = keys.primary_key
from predibase import Predictor
from predibase.pql import Session

predictor = Predictor(Session(token=AZURE_API_KEY), slugify(
predictor.http_endpoint = f"{ENDPOINT_NAME}"

preds = predictor.predict(df.head())
0False[0.9842461, 0.015753938]
1True[2.3841858e-07, 0.99999976]

Using Python standard library

Alternatively, we can use the Python standard library to make predictions.

import requests

data = {
"inputs": [
"name": "comment",
"shape": [1],
"datatype": "BYTES",
"data": ["I love this product"],
headers = {"Authorization": f"Bearer {AZURE_API_KEY}"}
url = f"https://{ENDPOINT_NAME}{slugify(}/infer"

response =, json=data, headers=headers)

Clean up

When you're done, remove the endpoint to avoid unexpected charges to your Azure account: