test_classification
Evaluate a pretrained classification model on custom test data.
Usually employed after the train_classification step. Useful to potentially
refit the model on a single dataset split and/or predict samples the model
hasn’t seen before, calculate errors etc., i.e. for sample-based analysis
of model performance.
Usage
The following examples show how the step can be used in a recipe.
Examples
Examples
Assuming we have reserved a test set containing data that wasn’t used to train the model, we can simply pass it to this step for evaluation:
Assuming we have reserved a test set containing data that wasn’t used to train the model, we can simply pass it to this step for evaluation:
If the test data is contained in a larger dataset (e.g. along training data), but can be identified using a column indicating the split, we can use the following setup:
Alternatively, we can create a randomized train/test split on the fly, re-fit the model on the train set, and evaluate on the test set. In this case an additional column will be added to the dataset, indicating the split each row belongs to:
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (ds.first_name), datasets (ds or ds[["first_name", "last_name"]]) or models (referenced
by name e.g. "churn-clf").
Inputs
Inputs
Outputs
Outputs
Column containing the model predictions.
Column containing the model prediction probabilities.
Column containing the model prediction errors.
Optional column identifying the train/test split, if dataset was randomly sampled and model re-fit.
Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last “input” to the step, i.e. step(..., {"param": "value", ...}) -> (output).
Parameters
Parameters
Target variable. Name of the column that contains your target values (labels).
Name of the positive class. In binary classification, usually the class you’re most interested in, for example the label/class corresponding to successful lead conversion in a lead score model, the class corresponding to a customer who has churned in a churn prediction model, etc.
If provided, will return predicted probabilities for the positive class. If not provided, will return probabilities for the predicted class (i.e. the class with the highest probability).
Train/test split configuration. Identify the splits using an existing column or create a randomized split. In either case, the model will be refit on the train split and evaluated on the test split.
Options
Options
Size of test split. The fraction of data used for testing. The remaining data will be used to refit the model.
Values must be in the following range:
Random seed. Seed used to initialize the random number generator assigning rows to train/test splits. If none is provided, result will be non-deterministic.
Size of test split. The fraction of data used for testing. The remaining data will be used to refit the model.
Values must be in the following range:
Random seed. Seed used to initialize the random number generator assigning rows to train/test splits. If none is provided, result will be non-deterministic.
Split column. Name of the column that contains the split identifiers/names.
Test split identifier.
Value of the split column that identifies the test set. Rows with this value will be used
to evaluate the model. If no train_split parameter is provided, the remaining rows will be
used to refit the model before evaluation.
Train split identifier. Value of the split column that identifies the train set. Rows with this value will be used to refit the model before evaluation. If not provided, all rows not belonging to the test split will be used in the refit.
Whether to retrain the model.
If set to true, the model will be refit on the train split before evaluation. If set to false,
the model will be evaluated on the test split without refitting. If no split configuration is provided,
this parameter is ignored.