Skip to main content

LLM Status

LLM deployments do not remain completely active all of the time and they can be in various operational statuses over the course of your workstream. These are a few methods available to check on your deployments or trigger them to get ready for prompting.

llm_deployment.get_events

llm_deployment.get_events()

This method returns fine-grained details about the state of your deployment in Predibase. This includes timestamped information about when the model was queued, the underlying hardware is being acquired or has been acquired, and when the deployment is running and ready to accept queries. Note that you can call this method even after the model has been deleted to retrieve information about your deployment.

Parameters:​

None

Returns:​

A table with time-stamped status updates, as well as how long the deployment has been/was active for.

Example Usage:​

Get the status of an LLM deployment (e.g. llm_deployment from the previous page) 

llm_deployment = pc.LLM("pb://deployments/deployment-name")
llm_deployment.get_events()

# output
Resource Name: deployment-name

Event Type    Timestamp
------------  ---------------------------
Pending       2023-12-07T13:18:59.885966Z
ScalingUp     2023-12-07T13:19:15.059688Z
Scheduled     2023-12-07T13:21:09.689012Z
Ready         2023-12-07T13:24:10.136454Z

Resource Active Time (seconds): 5765

llm_deployment.get_status

llm_deployment.get_status()

This method returns the state of your deployment job in Predibase.

Parameters:​

None

Returns:​

A short description of your deployment status, such as "queued", "updating", "active", etc.

Example Usage:​

Get the status of an LLM deployment (e.g. llm_deployment from the previous page) 

llm_deployment = pc.LLM("pb://deployments/deployment-name")
llm_deployment.get_status() # queued

llm_deployment.wait_for_ready

llm_deployment.wait_for_ready()

This method blocks until your deployment is ready for complete usage (i.e. both active and scaled up compute-wise).

Parameters:​

None

Returns:​

None

Example Usage:​

Get the status of an LLM deployment (e.g. llm_deployment from the previous page) 

llm_deployment = pc.LLM("pb://deployments/deployment-name")
llm_deployment.wait_for_ready()
# waits.
# waits..
# waits...
# return when ready!

For users looking for finer-grained control and insight into their deployments, there are two more specialized methods for assessing deployment state:

llm_deployment.is_ready

llm_deployment.is_ready()

This method returns whether your deployment is active and its endpoint is responsive

Parameters:​

None

Returns:​

True or False

Example Usage:​

llm_deployment = pc.LLM("pb://deployments/deployment-name")
llm_deployment.is_ready() # True or False