EVALUATE

EVALUATE is used to evaluate one or more target features from a trained model given a query input.

Syntax

Following is the grammar for making predictions:

EVALUATE target [, target ]* GIVEN query_input

target:
    feature_name [ WITH METADATA ] 
    [ USING model_name [ VERSION model_version] ]

query_input:
    SELECT feature_name [, feature_name]*
    FROM dataset_name
        [ WHERE where_condition ]
        [ ORDER BY sort_col { ASC | DESC } [ LIMIT row_count ] ]    

The * syntax in the above grammar indicates that an element can repeat zero or more times.

Target features

Evaluates begins with a series of target features for metrics to be calculated on.

With property

The WITH clause can specify additional properties for evaluation that are included as additional columns in the result set.

Following is the set of supported properties:

• METADATA: returns the model_name and model_version loaded for the target.

Using clause

You can evaluate a model with a specific model and version if desired.

EVALUATE feature_name USING model_name [ VERSION model_version]

By default, PQL will select the default model for all targets, but a specific model name and version can also be provided.

Query input

The test dataset is provide using a SQL select statements eg:

GIVEN SELECT * FROM dataset_name WHERE split=0 LIMIT 10;

Examples

Following are a list of example predict queries, of which could be composed into more complex SELECT statements.

Basic evaluation

The simplest form of the evaluation query.

EVALUATE Survived GIVEN SELECT * from titanic 

Compare evaluation

The WITH METADATA property can be used in conjunction with a UNION of two evaluation results to compare the default model for a given target vs version 1 of the titanic_small model as show in the following example which returns the model with the highest accuracy:

with results as (
  SELECT * FROM (
    EVALUATE Survived WITH METADATA
    GIVEN SELECT * from titanic 
  )
  UNION ALL SELECT * FROM (
    EVALUATE Survived WITH METADATA
    USING titanic_small VERSION 1 -- specific model
    GIVEN SELECT * from titanic
  )
)
SELECT * FROM results
WHERE metric_name = 'accuracy'
ORDER BY metric_value DESC LIMIT 1