Skip to main content

Quickstart

Predibase provides the fastest way to fine-tune and serve open-source LLMs. It's built on top of open-source LoRAX.

  • Inference: Try the Python SDK / REST or the Web Playground to prompt serverless endpoints
  • Fine-Tuning: Fine-tune and serve a model in just a few steps using the SDK or UI

Run inference using the SDK or REST​

  1. Create an account here.
  2. Navigate to the Settings page and click Generate API Token.
  3. Install the Python SDK with pip install -U predibase
  4. See available serverless deployments. (Note: VPC customers will need to first deploy a dedicated deployment.)
from predibase import Predibase, FinetuningConfig, DeploymentConfig

pb = Predibase(api_token="<PREDIBASE API TOKEN>")

lorax_client = pb.deployments.client("mistral-7b-instruct-v0-2") # Insert deployment name here
resp = lorax_client.generate("[INST] What are some popular tourist spots in San Francisco? [/INST]")
print(resp.generated_text)
info

Note the explicit use of special tokens before and after the prompt. These are used with instruction- and chat-tuned models to improve response quality. See Instruction Templates for details on how these should be applied for each of the serverless model endpoints.

Streaming​

from predibase import Predibase, FinetuningConfig, DeploymentConfig

pb = Predibase(api_token="<PREDIBASE API TOKEN>")

for resp in lorax_client.generate_stream("[INST] What are some popular tourist spots in San Francisco? [/INST]"):
    if not resp.token.special:
        print(resp.token.text, sep="", end="", flush=True)

Next steps​

  • Try out the full example to fine-tune and prompt an adapter in Predibase using the SDK
  • Don't want to code at all? Use the UI to connect a dataset and start fine-tuning an adapter.
  • Coming from OpenAI? Check out our migration guides for serving
  • Explore additional complete examples
  • See how you Predibase integrates with other frameworks in the ecosystem

Get in touch​

Reach out to us at support@predibase.com or join us on Discord for any questions, comments, or feedback!