Warning: This is a beta feature. APIs and behavior may change.Use the batch inference API to process inference requests in bulk at a lower cost.
Dataset Format
Batch inference takes in a dataset as input, with each row being used as a prompt to the model. The input dataset must conform to the OpenAI batch inference format. Example dataset row:model
field. The base model
will be specified when creating the batch job.
Once your JSONL file is ready,
upload it to Predibase.
For information about preparing your dataset, see
Dataset Preparation.
Creating a Batch Job
Warning: Currently only base models up to 16B parameters are supported.
Monitoring Jobs
Getting Results
Once the job status iscompleted
, download the results:
custom_id
field to match results with inputs.