filter_duplicates
Filter duplicate rows, keeping the first or last of each set of duplicates found only.
Usage
The following example shows how the step can be used in a recipe.
To keep only the first row amongst a set of duplicates, identifying duplicates by inspecting values in columns “address” and “name”
To keep only the first row amongst a set of duplicates, identifying duplicates by inspecting values in columns “address” and “name”
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (ds.first_name), datasets (ds or ds[["first_name", "last_name"]]) or models (referenced
by name e.g. "churn-clf").
Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last “input” to the step, i.e. step(..., {"param": "value", ...}) -> (output).
Names of columns used to detect and filter rows containing duplicate values. If not provided, will inspect all columns. Note that multivalued columns, i.e. those containing lists of values will always be ignored when searching for duplicates (but will be included in the result).
Which of a duplicate set of rows to keep in the result. Specifically, whether to keep the first or last row amongst the duplicates.
Values must be one of the following:
firstlast
if true, inverts the row selection.
I.e., only rows being duplicates (in the selected columns) will be included in the resulting dataset.
Row sorting before de-duplication. E.g. when the order of first or last duplicate to retain depends on other variables. If not configured, no sorting will be performed.