# embed_dataset
Source: https://docs.graphext.com/api-docs/analyse/embed/embed_dataset
Reduce the dataset to an n-dimensional numeric vector embedding.
Works like [`vectorize_dataset`](../vectorize_dataset/), but instead of converting the input dataset to a new dataset of
N numeric columns, it creates a single column in the original dataset containing vectors (lists) of N components. In other words,
the result of `vectorize_dataset` is converted to a column of embeddings, where each embedding is a numerical representation
of the corresponding row in the original dataset.
Many machine learning and AI algorithms expect their input data to be in pure numerical form, i.e. not containing categorical
variables, missing values etc. This step converts arbitrary datasets, potentially containing non-numerical variables and NaNs,
into this expected form. It does this by defining for each possible type of input column a transformation from non-numeric to
numeric values. As an example, ordered categorical variables (ordinals) such as the day of week, may be converted into a series
of numbers (0..7). Non-ordered categorical variables of low-cardinality (containing few different categories) may be expanded
into multiple new columns of 0s and 1s, indicating whether each row belongs to a specific category or not. Similar transformations
are applied to dates, multivalued categoricals etc.
NaNs are imputed (replaced) with an appropriate value from the corresponding column (e.g. the median in a quantitative column).
In addition, a new component of 0s and 1s is added, indicating whether the original column had a missing value or not.
The resulting embeddings will almost certainly not contain the same number of components as the original dataset's columns
(as the example of categorical variables shows).
The `n_components` parameter controls how many components the embeddings should have, and if this is smaller than would result
normally, a dimensionality reduction will be applied ([UMAP](https://umap-learn.readthedocs.io/en/latest/) by default).
The resulting numerical representation of the original data points aims to preserve the structure of similarities. I.e. if two
original rows are similar to each other, than their (potentially reduced) numerical representations should also be similar.
Equally, two very different rows should have representations that are also very different.
## Usage
The following examples show how the step can be used in a recipe.
The following, simplest, example, creates a new column of vector embeddings, each containing numeric components only, and hopefully capturing the same or most of the information in the corresponding original row.
```stan theme={null}
embed_dataset(ds) -> (ds.embedding)
```
The following example will convert and reduce the input dataset to a single column of embedding vectors (lists of numbers) each having 10 components. After normalization, the `date` component will be multiplied by 0.5 to reduces its weight relative to the others. The column `age` on the other hand will be given more importance. Also, 15 neighbours are considered for each data point in UMAP, so that we give more importance to the similarity between nearby points and less importance to the global structure of the data when calculating the embeddings.
```stan theme={null}
embed_dataset(ds, {
"weights": {"date": 0.5, "age": 2},
"weights_max": 32,
"weights_exp": 2,
"algorithm": "umap",
"n_components": 10,
"n_neighbors": 15,
"min_dist": 0.1,
"random_state": 42
}) -> (ds_vec)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
embed_dataset(ds: dataset, {
"param": value,
...
}) -> (embedding: list[number], *links: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An arbitrary input dataset.
A column containing embedding vectors (lists of numbers) numerically representing the correspoding rows in `ds`.
Two optional columns holding the links between nearest neighbours in space of the created embeddings.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Algorithm.
The name of a supported dimensionality reduction algorithm.
Values must be one of the following:
* `umap`
Toggle encoding of feature columns.
When enabled, Graphext will auto-convert any column types to the numeric type before
(optionally) reducing the data's dimensionality. How this conversion is done can be
configured using the `feature_encoder` option below.
If disabled, the dimensionality reduction algorithm applied in this step will
assume that input data is already numerical and doesn't contain any missing
values.
Configures encoding of feature columns.
By default (`null`), Graphext chooses automatically how to convert any column types the model
may not understand natively to a numeric type.
A configuration object can be passed instead to overwrite specific parameter values with respect
to their default values.
Numeric encoder.
Configures encoding of numeric features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Further parameters passed to the `scaler` function.
Details depend no the particular scaler used.
Boolean encoder.
Configures encoding of boolean features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Ordinal encoder.
Configures encoding of categorical features that have a natural order.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Category encoder.
May contain either a single configuration for all categorical variables, or two different configurations
for low- and high-cardinality variables. For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for categories with fewer than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Multilabel encoder.
Configures encoding of multivalued categorical features (variable length lists of categories,
or the semantic type `list[category]` for short). May contain either a single configuration for
all multilabel variables, or two different configurations for low- and high-cardinality variables.
For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for mulitabel columns with fewer than `cardinality_threshold` unique categories/labels.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Datetime encoder.
Configures encoding of datetime (timestamp) features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
A list of numerical components to extract.
Will create one numeric column for each component.
Each item in array.
Values must be one of the following:
`day` `dayofweek` `dayofyear` `hour` `minute` `month` `quarter` `season` `second` `week` `weekday` `weekofyear` `year`
A list of cyclical time features to extract.
"Cycles" are numerical transformations of features that should be represented on a circle. E.g. months,
ranging from 1 to 12, should be arranged such that 12 and 1 are next to each other, rather than on
opposite ends of a linear scale. We represent such cyclical time features on a circle by creating two
columns for each original feature: the sin and cos of the numerical feature after appropriate scaling.
Each item in array.
Values must be one of the following:
* `day`
* `dayofweek`
* `dayofyear`
* `hour`
* `month`
Whether to include the epoch as new feature (seconds since 01/01/1970).
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Embedding/vector encoder.
Configures encoding of multivalued numerical features (variable length lists of numbers, i.e. vectors, or the semantic type `list[number]` for short).
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Text encoder.
Configures encoding of text (natural language) features. Currently only allows
[Tf-Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) embeddings to represent texts. If you wish
to use other embeddings, e.g. semantic, Word2Vec etc., transform your text column first using
another step, and then use that result instead of the original texts.
Texts are *excluded* by default from the overall encoding of the dataset. See parameter
`include_text_features` below to active it.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Parameters to be passed to the text encoder (Tf-Idf parameters only for now).
See [scikit-learn's documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)
for detailed parameters and their explanation.
How many output features to generate.
The resulting Tf-Idf vectors will be reduced to these many dimensions (columns) using scikit-learn's
[truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
This performs a kind of [latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
By default we will reduce to 200 components.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_components ≤ 1024
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Whether to include or ignore text columns during the processing of input data.
Enabling this will convert texts to their Tf-Idf representation. Each text will be
converted to an N-dimensional vector in which each component measures the relative
"over-representation" of a specific word (or n-gram) relative to its overall
frequency in the whole dataset. This is disabled by default because it will
often be better to convert texts explicitly using a previous step, such as
`embed_text` or `embed_text_with_model`.
Weights used to multiply the normalized columns/features after vectorization.
Should be a dictionary/object of `{"column_name": weight, ...}` items. Will be scaled using the
parameters `weights_max`, and `weights_exp` before being applied. So only the relative weight of
the columns is important here, not their absolute values.
A `"column_name": numeric_weight` pair.
Each column name must refer to an existing column in the dataset.
* `{"date": 0.5, "age": 2}`
Weights used to multiply the normalized columns/features after vectorization.
Should be a dictionary/object of `"type": weight"` items. Will be scaled using the parameters
`weights_max`, and `weights_exp` before being applied. So only the relative weight of the columns
is important here, not their absolute values.
Weight for columns of type `Number`
Weight for columns of type `Datetime`
Weight for columns of type `Category`
Weight for columns of type `Ordinal`
Weight for columns of type `Embedding` (`List[Number]`).
Weight for columns of type `Multilabel` (`List[Category]`).
Maximum weight to scale the normalized columns with.
Values must be in the following range:
```javascript theme={null}
0 ≤ weights_max < inf
```
Weight exponent.
Weights will be raised to this power before(!) scaling to `weights_max`. This allows for a non-linear
mapping from input weights to those used eventually to multiply the normalized columns.
Number of neighbours.
Use smaller numbers to concentrate on the local structure in the data, and larger values to focus on the
more global structure.
For further details see [here](https://umap-learn.readthedocs.io/en/latest/parameters.html#n-neighbors).
Values must be in the following range:
```javascript theme={null}
1 ≤ n_neighbors < inf
```
Minimum distance between reduced data points.
Controls how tightly UMAP is allowed to pack points together in the reduced space. Smaller values will
lead to points more tightly packed together (potentially useful if result is used to cluster the points).
Larger values will distribute points with more space between them (which may be desirable for visualization,
or to focus more on the global structure of the date).
For further details see [here](https://umap-learn.readthedocs.io/en/latest/parameters.html#min-dist).
Values must be in the following range:
```javascript theme={null}
0 ≤ min_dist < inf
```
Dimensionality of the reduced data.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_components < inf
```
Metric to use for measuring similarity between data points.
Values must be one of the following:
`euclidean` `manhattan` `chebyshev` `minkowski` `canberra` `braycurtis` `haversine` `mahalanobis` `wminkowski` `seuclidean` `cosine` `correlation` `hamming` `jaccard` `dice` `russellrao` `kulsinski` `rogerstanimoto` `sokalmichener` `sokalsneath` `yule`
Number of training iterations used in optimizing the embedding.
Larger values result in more accurate embeddings. If `null` is specified a value will be
selected based on the size of the input dataset (200 for large datasets, 500 for small).
How to initialize the low dimensional embedding.
When "spectral", uses a (relatively expensive) spectral embedding. "pca" uses the first `n_components`
from a principal component analysis. "tswspectral" is a cheaper alternative to "spectral". When "random",
assigns initial embedding positions at random. This uses the least amount of memory and time but may make UMAP
slower to converge on the optimal embedding. "auto" selects between "spectral" and "random" automatically
depending on the size of the dataset.
Values must be one of the following:
* `spectral`
* `pca`
* `tswspectral`
* `random`
* `auto`
Avoid excessive memory use.
For some datasets nearest neighbor computations can consume a lot of memory. If you find
the step is failing due to memory constraints, consider setting this option to `true`.
This approach is more computationally expensive, but avoids excessive memory use. Setting
it to "auto", will enable this mode automatically depending on the size of the dataset.
Values must be one of the following:
* `True`
* `False`
* `auto`
* `None`
Target variable (labels) for supervised dimensionality reduction.
Name of the column that contains your target values (labels).
Weighting factor between features and target.
A value of 0.0 weights entirely on data, and a value of 1.0 weights entirely on target. The
default of 0.5 balances the weighting equally between data and target.
Try to better preserve local densities in the data.
Specifies whether the density-augmented objective of densMAP should be used for optimization.
Turning on this option generates an embedding where the local densities are encouraged to be
correlated with those in the original space.
Strength of local density preservation.
Controls the regularization weight of the density correlation term in densMAP. Higher values
prioritize density preservation over the UMAP objective, and vice versa for values closer to zero.
Setting this parameter to zero is equivalent to running the original UMAP algorithm.
Drop duplicate rows before embedding.
If you have more duplicates than you have `n_neighbors` you can have the identical data points lying
in different regions of your space. It also violates the definition of a metric. This option will
remove duplicates before embedding, and then map the original data points back to the reduced space. Duplicate
data points will be placed in the exact same location as the original data points.
A random number to initialize the algorithm for reproducibility.
Maintain links for n nearest neighbours only in graph.
Values must be in the following range:
```javascript theme={null}
1 ≤ links_top_n ≤ 15
```
# embed_images
Source: https://docs.graphext.com/api-docs/analyse/embed/embed_images
Embed images using pretrained DL models.
An embedding vector is a numerical representation of an image (or text etc.), such that different numerical components
of the vector capture different dimensions of the image's content. Embeddings can be used, for example, to calculate
the *semantic similarity* between pairs of images (see `link_embeddings`, for example, to create a network of images
connected by similarity).
In its current form the step calculates image embeddings using [Clip](https://huggingface.co/docs/transformers/model_doc/clip),
which has been trained on 400M image/text pairs to pick out an image's correct caption from a list of candidates.
## Usage
The following example shows how the step can be used in a recipe.
The step has no required parameters, so the simplest call is simply
```stan theme={null}
embed_images(ds.image_url) -> (ds.embedding)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
embed_images(images: url, {
"param": value,
...
}) -> (embedding: list[number])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column of URLs to images to calculate embeddings for.
A column of embedding vectors capturing the meaning of each input image.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Whether to normalize embedding vectors (to length/norm of 1.0).
# embed_items
Source: https://docs.graphext.com/api-docs/analyse/embed/embed_items
Trains an _item2vec_ model on provided lists of items (or sentences of words, etc.).
This is essentially the [*word2vec*](https://en.wikipedia.org/wiki/Word2vec) algorithm applied to arbitrary lists
of items. *Word2vec* computes vectors representing words such that nearby (similar) vectors represent words that are
often found in a similar context. *Item2vec* refers to using the exact same algorithm but applying it to arbitrary
lists of items in which the order of items has a comparable interpretation to words in a sentence (the items may be
categories, tags, IDs etc.).
Note, that if the order of items in the list (session/basket etc.) is not important, and you simply want item vectors
to be similar if the corresponding items usually occur together in the same list, use the `window` parameter (see
below) with a value of "all".
We use [gensim](https://radimrehurek.com/gensim/) to train the *item2vec* model, so for further details also see it's
[word2vec page](https://radimrehurek.com/gensim/models/word2vec.html).
## Usage
The following example shows how the step can be used in a recipe.
The following uses default parameter values only, and thus would be equivalent to using the step without specifying
any parameters.
```stan theme={null}
embed_items(products.id, baskets.product_ids, {
"size": 48,
"sg": 1,
"negative": 20,
"alpha": 0.025,
"window": 5,
"min_count": 3,
"iter": 10,
"sample": 0
}) -> (products.embedding)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
embed_items(items: category|number, sessions: list[category]|list[number], {
"param": value,
...
}) -> (embeddings: list[number])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column containing item identifiers (IDs).
A column containing lists, where each row is a session, and each session a list of item identifiers (IDs) compatible
with the values of the items column.
A list column containing item embeddings in the same order as the items input column. Embeddings are lists of numbers
(vectors).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Length of resulting embedding vectors.
Values must be in the following range:
```javascript theme={null}
1 ≤ size < inf
```
Whether to use the skip-gram or CBOW algorithm.
Set this to 1 for skip-gram, and 0 for CBOW.
Values must be in the following range:
```javascript theme={null}
0 ≤ sg ≤ 1
```
Update maximum for negative-sampling.
Only update these many word vectors.
Initial learning rate.
Values must be in the following range:
```javascript theme={null}
0 ≤ alpha ≤ 1
```
Size of word context window.
Must be either an integer (the number of neighbouring words to consider), or any of "auto", "max" or "all",
in which case the window is equal to the whole list/session/basket.
integer.
Values must be in the following range:
```javascript theme={null}
1 ≤ {_} < inf
```
string.
Values must be one of the following:
* `auto`
* `max`
* `all`
Minimum count of item in dataset.
If an item occurs fewer than this many times it will be ignored.
Values must be in the following range:
```javascript theme={null}
1 ≤ min_count < inf
```
Iterations.
How many epochs to run the algorithm for.
Values must be in the following range:
```javascript theme={null}
1 ≤ iter < inf
```
Percentage of most-common items to filter out (equivalent to "stop words").
Values must be in the following range:
```javascript theme={null}
0 ≤ sample ≤ 1
```
Whether to return normalized item vectors.
# embed_sessions
Source: https://docs.graphext.com/api-docs/analyse/embed/embed_sessions
Trains an _item2vec_ model on provided lists of items.
Lists of items may represent pages visited in a browsing session, shopping baskets and the products they contain,
sentences of words, etc. The step calculates embeddings vectors for all item lists, such that two vectors are similar
if their corresponding lists of items are similar. Similarity here is measured as an average over the individual
items. Essentially, we first calculate embeddings vectors representing individual items (using [word2vec](https://en.wikipedia.org/wiki/Word2vec)),
and then average over all items belonging to the same list/session.
As an example, consider a dataset containing shopping baskets. In this case the step will first calculate embeddings
for individual products. The resulting vectors will be similar if they represent objects that are often bought together.
E.g. the vectors for sausages and hot dog bread may be more similar to each other than those representing shampoo and
toys. Then, to arrive at an embedding vector for each basket, we simply average over all its individual products. The result
will capture the similarity between baskets in terms of the mix of products they contain. And so the vectors representing
baskets of people buying a significant amount of baby products will be more similar to each other than to vectors representing
baskets of people buying products for a BBQ party.
To only calculate individual item embeddings see the complementary [embed\_items](../embed_items/) step.
Also, we use [gensim](https://radimrehurek.com/gensim/) to train the *item2vec* model, so for further details also see it's
[word2vec page](https://radimrehurek.com/gensim/models/word2vec.html).
## Usage
The following example shows how the step can be used in a recipe.
```stan theme={null}
embed_sessions(baskets.products, {
"size": 48,
"sg": 1,
"negative": 20,
"alpha": 0.025,
"window": 5,
"min_count": 3,
"iter": 10,
"sample": 0
}) -> (baskets.embedding)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
embed_sessions(sessions: list[category]|list[number], {
"param": value,
...
}) -> (embeddings: list[number])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column containing lists, where each row is a session, and each session a list of items.
A column containing item embeddings in the same order as the items input column. Embeddings are lists of numbers.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Length of embedding vectors.
Values must be in the following range:
```javascript theme={null}
1 ≤ size < inf
```
Use Skip-Gram or CBOW.
Set this to 1 to use Skip-Gram, 0 for CBOW.
Values must be in the following range:
```javascript theme={null}
0 ≤ sg ≤ 1
```
Update maximum for negative-sampling.
Only update these many word vectors.
Initial Learning Rate.
Values must be in the following range:
```javascript theme={null}
0 ≤ alpha ≤ 1
```
Word context window.
Must be either an integer or "auto", "max" or "all".
integer.
Values must be in the following range:
```javascript theme={null}
1 ≤ {_} < inf
```
string.
Values must be one of the following:
* `auto`
* `max`
* `all`
Minimum count of item in dataset. Otherwise filtered out.
Values must be in the following range:
```javascript theme={null}
1 ≤ min_count < inf
```
Iterations.
How many epochs to run the algorithm for.
Values must be in the following range:
```javascript theme={null}
1 ≤ iter < inf
```
Sample.
Percentage of most-common items to filter out.
Values must be in the following range:
```javascript theme={null}
0 ≤ sample ≤ 1
```
Whether to return normalized item vectors.
# embed_text
Source: https://docs.graphext.com/api-docs/analyse/embed/embed_text
Parse and calculate a (word-averaged) embedding vector for each text.
An embedding vector is a numerical representation of a text, such that different numerical components of the vector
capture different dimensions of the text's meaning. Embeddings can be used, for example, to calculate the *semantic similarity*
between pairs of texts (see `link_embeddings`, for example, to create a network of texts connected by similarity).
In this step, embeddings of texts are calculated as (weighted) averages of the embeddings of each text's individual
words (the individual word embeddings are [GloVe](https://nlp.stanford.edu/projects/glove/) vectors, as provided by
[spaCy's](https://spacy.io/) [language models](https://spacy.io/models/en#en_core_web_md)).
Use either the `language` *parameter* or a second input *column* to specify the language of the input texts. If neither
is provided, the language will be inferred automatically from the texts themselves (which is equivalent to first creating
a language column using the `infer_language` step).
## Usage
The following example shows how the step can be used in a recipe.
To calculate embeddings in a way that emphasizes entities (recognized products, people etc.) over regular words:
```stan theme={null}
embed_text(ds.text, ds.lang, {"weighted": true}) -> (ds.embedding)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
embed_text(text: text, *lang: category, {
"param": value,
...
}) -> (embedding: list[number])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A text column to calculate embeddings for.
An (optional) column identifying the languages of the corresponding texts. It is used to identify the correct model (spaCy)
to use for each text. If the dataset doesn't contain such a column yet, it can be created using the `infer_language` step.
Ideally, languages should be expressed as two-letter
[ISO 639-1 language codes](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes), such as "en", "es" or "de" for
English, Spanish or German respectively. We also detect fully spelled out names such as "english", "German", "allemande"
etc., but it is not guaranteed that we will recognize all possible spellings correctly always, so ISO codes should be
preferred.
Alternatively, if all texts are in the same language, it can be identified with the `language` *parameter* instead.
A column of embedding vectors capturing the meaning of each input text.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Configure how embeddings are calculated.
Toggle word vector weighting and normalization.
Whether entities have more influence on the embedding than regular words.
Whether to normalize embeddings. Each will have a length (norm) of 1.0.
Whether to enable support for additional languages.
By default, Arabic ("ar"), Catalan ("ca"), Basque ("eu"), and Turkish ("tu") are not enabled,
since they're supported only by a different class of language models (stanfordNLP's Stanza)
that is much slower than the rest. This parameter can be used to enable them.
Minimum number (or proportion) of texts to include a language in processing.
Any texts in a language with fewer documents than these will be ignored. Can be useful to speed up
processing when there is noise in the input languages, and when ignoring languages with a small number of
documents only is acceptable. Values smaller than 1 will be interpreted as a *proportion* of all texts, and
values greater than or equal to 1 as an *absolute number* of documents.
number.
Values must be in the following range:
```javascript theme={null}
0 < {_} < 1
```
integer.
Values must be in the following range:
```javascript theme={null}
1 ≤ {_} < inf
```
The language of inputs texts.
If all texts are in the same language, it can be specified here instead of passing it as an input column. The language will be used to identify the correct spaCy model to parse and analyze the texts. For allowed values, see the comment regarding the `lang` column above.
# embed_text_with_model
Source: https://docs.graphext.com/api-docs/analyse/embed/embed_text_with_model
Use language models to calulate an embedding for each text in provided column.
An embedding vector is a numerical representation of a text, such that different numerical components of the vector
capture different dimensions of the text's meaning. Embeddings can be used, for example, to calculate the *semantic similarity*
between pairs of texts. See [`link_embeddings`](https://docs.graphext.com/api-docs/analyse/graph_and_map/create_graph/link_embeddings/),
for example, to create a network of texts connected by similarity.
In this step, embeddings of texts are calculated using pre-trained
[neural language models](https://en.wikipedia.org/wiki/Language»model#Neural_network), especially those using the
popular [transformer architecture](https://huggingface.co/course/chapter1/4) (e.g.
[Bert-based models](https://huggingface.co/transformers/model_doc/bert.html)).
## Things to keep in mind
* Unlike [`embed_text`](https://docs.graphext.com/api-docs/prepare/embed/embed_text/), which uses a different, appropriate spaCy
model for each language in the text column, this step will always use a single model only to calculate embeddings. This
means the model should be multilingual if you have mixed languages, and that otherwise you need to choose the
correct model for your (single) language.
* Each model will be downloaded on the fly before processing the text. This adds a little lag to its execution time (the
bigger the model the longer the download), though for a sufficient number of texts the time spent downloading should not
be significant. Note also, however, that the download, and therefore this step, may fail if the servers of its publisher
are not responsive.
* Since this step potentially supports tens if not hundreds of different models, we cannot provide support or advice on
specific models.
## Usage
The following example shows how the step can be used in a recipe.
To calculate embeddings using a multilingual sentence-bert model (from sentence-transformers):
```stan theme={null}
embed_text_with_model(ds.text, {"collection": "SBERT", "name": "distiluse-base-multilingual-cased-v2"}) -> (ds.embedding)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
embed_text_with_model(text: text, {
"param": value,
...
}) -> (embedding: list[number])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A text column to calculate embeddings for.
A column of embedding vectors capturing the meaning of each input text.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Embed texts using a *Sentence-BERT* model.
Models in this collection (also known as *sentence-transformers*) have been trained specifically for semantic
similarity, i.e. for the purpose of comparing the meaning of texts. Individual models in this collection
can be found here: [https://www.sbert.net/docs/pretrained\_models.html](https://www.sbert.net/docs/pretrained_models.html).
They differ in terms of the language they have been trained on; their size (the bigger the better usually,
but also the slower); as well as their purpose or intended area of application (e.g. it has a specific model
to embed scientific publications).
A specific *Sentence-BERT* model name.
To find a model appropriate for your data or task, check the website of the
[Sentence-BERT model collection](https://www.sbert.net/docs/pretrained_models.html).
* paraphrase-MiniLM-L6-v2
* distiluse-base-multilingual-cased-v2
Whether text embedding vectors should be normalized (to lengths of 1.0).
This may make similarity calculations easier. E.g. we can then use the dot product as a similarity "metric",
instead of the usual cosine angle (which not all downstream functions may support).
How many texts to push through the model at the same time.
Greater values usually mean faster processing (if supported by the model), but also greater use of memory.
Values must be in the following range:
```javascript theme={null}
1 ≤ batch_size < inf
```
Embed texts using a model from the *Hugging Face* hub.
Any pytorch or tensorflow model in [HF's hub](https://huggingface.co/models)
can be used as long as its output contains a [last hidden state](https://huggingface.co/transformers/main_classes/output.html#).
Note however, that using the output embedding of an arbitrary transformer is not always useful, and
specifically may not be approriate for sentence similarity. Rather, these embeddings usually represent the
input for downstream classification tasks instead. A sentence-bert or universal sentence encoder may be more
appopriate in most cases.
A specific *Hugging Face* model name.
To find a model appropriate for your data or task, browse the [Hugging Face model hub](https://huggingface.co/models).
Note that the `name` of a model should include the name of the organization if applicable (e.g.
`"cardiffnlp/"` in the example below).
* cardiffnlp/twitter-xlm-roberta-base
* sentence-transformers/paraphrase-xlm-r-multilingual-v1
Whether text embedding vectors should be normalized (to lengths of 1.0).
This may make similarity calculations easier. E.g. we can then use the dot product as a similarity "metric",
instead of the usual cosine angle (which not all downstream functions may support).
How many texts to push through the model at the same time.
Greater values usually mean faster processing (if supported by the model), but also greater use of memory.
Values must be in the following range:
```javascript theme={null}
1 ≤ batch_size < inf
```
How individual "word" embeddings should be combined.
The output of a transformer contains embeddings for individual words (or sentence pieces, sub-word character
sequences etc.). This parameter determines how these are combined to create a single vector representing the
whole text. This can be the *mean* of individual vectors or the (component-wise) *maximum* (currently pooling
doesn't take the attention mask into account).
Values must be one of the following:
* `mean`
* `max`
# embed_with_trees
Source: https://docs.graphext.com/api-docs/analyse/embed/embed_with_trees
Reduce the dataset to an n-dimensional numeric vector embedding using a Forest model's tree indices.
Usually employed after the `train_classifcation` or `train_regression` steps with RandomForest/ExtraTrees/Catboost models.
???+ info "Prediction Model"
To use this step successfully you need to make sure the dataset you're predicting on is
as similar as possible to the one the model was trained on. We check that the necessary data
types and columns are present, but you should pay attention to how you handled these in the
recipe the model was generated. Any changes might lead to a significant degradation in
model performance.
This process needs a pre-trained RandomForest, ExtraTrees or Catboost model trained through the `train_classification` or `train_regression` methods on the same dataset that is used as input.
By calling this method, each data point in the dataset is passed through the trees in the forest,
and the leaf node where each data point ends up in each tree is recorded.
The indices of these leaf nodes across all trees in the forest are then used to form a sparse
high-dimensional representation of each data point. This representation can be thought of as an embedding,
where the position of each data point in this high-dimensional space captures aspects of its similarity
to other data points, as determined by the structure of the trees in the model.
For more detailed information on the method followed, you can check the `apply` method of sklearns' forest classes
and its usage [here](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier.apply).
## Usage
The following example shows how the step can be used in a recipe.
The following, simplest, example, creates a new embedding column from a dataset and a RandomForest model.
```stan theme={null}
embed_with_trees(ds, "rf") -> (ds.embedding)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
embed_with_trees(ds: dataset, model: file
) -> (embedding: list[number])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An arbitrary input dataset.
A trained model of the RandomForest, ExtraTrees or Catboost kind.
A column containing embedding vectors (lists of numbers) numerically representing the correspoding rows in `ds`.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
This step doesn't expect any configuration.
# Embed
Source: https://docs.graphext.com/api-docs/analyse/embed/index
| Step | Fast | Description |
| ------------------------------------------------------------------------- | ---- | --------------------------------------------------------------------------------------------------- |
| [embed\_dataset](/api-docs/analyse/embed/embed_dataset) | | Reduce the dataset to an n-dimensional numeric vector embedding |
| [embed\_images](/api-docs/analyse/embed/embed_images) | | Embed images using pretrained DL models |
| [embed\_items](/api-docs/analyse/embed/embed_items) | | Trains an *item2vec* model on provided lists of items (or sentences of words, etc.) |
| [embed\_sessions](/api-docs/analyse/embed/embed_sessions) | | Trains an *item2vec* model on provided lists of items |
| [embed\_text](/api-docs/analyse/embed/embed_text) | | Parse and calculate a (word-averaged) embedding vector for each text |
| [embed\_text\_with\_model](/api-docs/analyse/embed/embed_text_with_model) | | Use language models to calulate an embedding for each text in provided column |
| [embed\_with\_trees](/api-docs/analyse/embed/embed_with_trees) | | Reduce the dataset to an n-dimensional numeric vector embedding using a Forest model's tree indices |
| [layout\_dataset](/api-docs/analyse/embed/layout_dataset) | | Reduce the dataset to 2 dimensions that can be mapped to x/y node positions |
| [vectorize\_dataset](/api-docs/analyse/embed/vectorize_dataset) | | Create a vectorized (numeric) dataset, (optionally) of reduced dimensionality |
# layout_dataset
Source: https://docs.graphext.com/api-docs/analyse/embed/layout_dataset
Reduce the dataset to 2 dimensions that can be mapped to x/y node positions.
Based on the same vectorization and dimensionality reduction as the steps [`vectorize_dataset`](https://docs.graphext.com/api-docs/prepare/embed/vectorize_dataset/)
and [`embed_dataset`](https://docs.graphext.com/api-docs/prepare/embed/embed_dataset/). The only difference being that the number of dimensions (output columns) is
fixed to 2 (corresponding to x and y positions).
## Usage
The following examples show how the step can be used in a recipe.
The following, simplest, example, will convert the input dataset `ds` to purely numerical form, will reduce its dimensionality to just 2 using [UMAP](https://umap-learn.readthedocs.io/en/latest/), and will save those 2 dimensions in the columns `x` and `y`. The way that the x and y coordinates are calculated via dimensionality reduction should preserve the similarity between original rows. I.e., rows that are similar in the original dataset should have coordinates close to each other.
```stan theme={null}
layout_dataset(ds) -> (ds.x, ds.y)
```
The following example will numerically convert and reduce the input dataset to 2 dimensions: x and y. 15 neighbours are considered for each data point during dimensionality reduction (instead of the default 100), so that we give more importance to the similarity between nearby points and less importance to the global structure of the data when calculating the layout. Also, the minimum distance of neighbours is increased from 0.1 to 0.8, to create a less dense and overlapping layout.
```stan theme={null}
layout_dataset(ds, {
"algorithm": "umap",
"n_neighbors": 15,
"min_dist": 0.8,
"random_state": 42,
"scale": 200,
}) -> (ds.x, ds.y)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
layout_dataset(ds: dataset, {
"param": value,
...
}) -> (x: column, y: column, *links: column
)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An arbitrary input dataset.
Column containing the x coordinate for each row.
Column containing the y coordinate for each row.
Two optional columns holding the links between nearest neighbours in space of the created layout.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Algorithm.
The name of a supported dimensionality reduction algorithm.
Values must be one of the following:
* `umap`
Toggle encoding of feature columns.
When enabled, Graphext will auto-convert any column types to the numeric type before
(optionally) reducing the data's dimensionality. How this conversion is done can be
configured using the `feature_encoder` option below.
If disabled, the dimensionality reduction algorithm applied in this step will
assume that input data is already numerical and doesn't contain any missing
values.
Configures encoding of feature columns.
By default (`null`), Graphext chooses automatically how to convert any column types the model
may not understand natively to a numeric type.
A configuration object can be passed instead to overwrite specific parameter values with respect
to their default values.
Numeric encoder.
Configures encoding of numeric features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Further parameters passed to the `scaler` function.
Details depend no the particular scaler used.
Boolean encoder.
Configures encoding of boolean features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Ordinal encoder.
Configures encoding of categorical features that have a natural order.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Category encoder.
May contain either a single configuration for all categorical variables, or two different configurations
for low- and high-cardinality variables. For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for categories with fewer than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Multilabel encoder.
Configures encoding of multivalued categorical features (variable length lists of categories,
or the semantic type `list[category]` for short). May contain either a single configuration for
all multilabel variables, or two different configurations for low- and high-cardinality variables.
For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for mulitabel columns with fewer than `cardinality_threshold` unique categories/labels.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Datetime encoder.
Configures encoding of datetime (timestamp) features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
A list of numerical components to extract.
Will create one numeric column for each component.
Each item in array.
Values must be one of the following:
`day` `dayofweek` `dayofyear` `hour` `minute` `month` `quarter` `season` `second` `week` `weekday` `weekofyear` `year`
A list of cyclical time features to extract.
"Cycles" are numerical transformations of features that should be represented on a circle. E.g. months,
ranging from 1 to 12, should be arranged such that 12 and 1 are next to each other, rather than on
opposite ends of a linear scale. We represent such cyclical time features on a circle by creating two
columns for each original feature: the sin and cos of the numerical feature after appropriate scaling.
Each item in array.
Values must be one of the following:
* `day`
* `dayofweek`
* `dayofyear`
* `hour`
* `month`
Whether to include the epoch as new feature (seconds since 01/01/1970).
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Embedding/vector encoder.
Configures encoding of multivalued numerical features (variable length lists of numbers, i.e. vectors, or the semantic type `list[number]` for short).
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Text encoder.
Configures encoding of text (natural language) features. Currently only allows
[Tf-Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) embeddings to represent texts. If you wish
to use other embeddings, e.g. semantic, Word2Vec etc., transform your text column first using
another step, and then use that result instead of the original texts.
Texts are *excluded* by default from the overall encoding of the dataset. See parameter
`include_text_features` below to active it.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Parameters to be passed to the text encoder (Tf-Idf parameters only for now).
See [scikit-learn's documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)
for detailed parameters and their explanation.
How many output features to generate.
The resulting Tf-Idf vectors will be reduced to these many dimensions (columns) using scikit-learn's
[truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
This performs a kind of [latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
By default we will reduce to 200 components.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_components ≤ 1024
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Whether to include or ignore text columns during the processing of input data.
Enabling this will convert texts to their Tf-Idf representation. Each text will be
converted to an N-dimensional vector in which each component measures the relative
"over-representation" of a specific word (or n-gram) relative to its overall
frequency in the whole dataset. This is disabled by default because it will
often be better to convert texts explicitly using a previous step, such as
`embed_text` or `embed_text_with_model`.
Weights used to multiply the normalized columns/features after vectorization.
Should be a dictionary/object of `{"column_name": weight, ...}` items. Will be scaled using the
parameters `weights_max`, and `weights_exp` before being applied. So only the relative weight of
the columns is important here, not their absolute values.
A `"column_name": numeric_weight` pair.
Each column name must refer to an existing column in the dataset.
* `{"date": 0.5, "age": 2}`
Weights used to multiply the normalized columns/features after vectorization.
Should be a dictionary/object of `"type": weight"` items. Will be scaled using the parameters
`weights_max`, and `weights_exp` before being applied. So only the relative weight of the columns
is important here, not their absolute values.
Weight for columns of type `Number`
Weight for columns of type `Datetime`
Weight for columns of type `Category`
Weight for columns of type `Ordinal`
Weight for columns of type `Embedding` (`List[Number]`).
Weight for columns of type `Multilabel` (`List[Category]`).
Maximum weight to scale the normalized columns with.
Values must be in the following range:
```javascript theme={null}
0 ≤ weights_max < inf
```
Weight exponent.
Weights will be raised to this power before(!) scaling to `weights_max`. This allows for a non-linear
mapping from input weights to those used eventually to multiply the normalized columns.
Number of neighbours.
Use smaller numbers to concentrate on the local structure in the data, and larger values to focus on the
more global structure.
For further details see [here](https://umap-learn.readthedocs.io/en/latest/parameters.html#n-neighbors).
Values must be in the following range:
```javascript theme={null}
1 ≤ n_neighbors < inf
```
Minimum distance between reduced data points.
Controls how tightly UMAP is allowed to pack points together in the reduced space. Smaller values will
lead to points more tightly packed together (potentially useful if result is used to cluster the points).
Larger values will distribute points with more space between them (which may be desirable for visualization,
or to focus more on the global structure of the date).
For further details see [here](https://umap-learn.readthedocs.io/en/latest/parameters.html#min-dist).
Values must be in the following range:
```javascript theme={null}
0 ≤ min_dist < inf
```
Dimensionality of the reduced data. Fixed at 2 for the purpose of a layout's x and y coordinates.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_components < inf
```
Metric to use for measuring similarity between data points.
Values must be one of the following:
`euclidean` `manhattan` `chebyshev` `minkowski` `canberra` `braycurtis` `haversine` `mahalanobis` `wminkowski` `seuclidean` `cosine` `correlation` `hamming` `jaccard` `dice` `russellrao` `kulsinski` `rogerstanimoto` `sokalmichener` `sokalsneath` `yule`
Number of training iterations used in optimizing the embedding.
Larger values result in more accurate embeddings. If `null` is specified a value will be
selected based on the size of the input dataset (200 for large datasets, 500 for small).
How to initialize the low dimensional embedding.
When "spectral", uses a (relatively expensive) spectral embedding. "pca" uses the first `n_components`
from a principal component analysis. "tswspectral" is a cheaper alternative to "spectral". When "random",
assigns initial embedding positions at random. This uses the least amount of memory and time but may make UMAP
slower to converge on the optimal embedding. "auto" selects between "spectral" and "random" automatically
depending on the size of the dataset.
Values must be one of the following:
* `spectral`
* `pca`
* `tswspectral`
* `random`
* `auto`
Avoid excessive memory use.
For some datasets nearest neighbor computations can consume a lot of memory. If you find
the step is failing due to memory constraints, consider setting this option to `true`.
This approach is more computationally expensive, but avoids excessive memory use. Setting
it to "auto", will enable this mode automatically depending on the size of the dataset.
Values must be one of the following:
* `True`
* `False`
* `auto`
* `None`
Target variable (labels) for supervised dimensionality reduction.
Name of the column that contains your target values (labels).
Weighting factor between features and target.
A value of 0.0 weights entirely on data, and a value of 1.0 weights entirely on target. The
default of 0.5 balances the weighting equally between data and target.
Try to better preserve local densities in the data.
Specifies whether the density-augmented objective of densMAP should be used for optimization.
Turning on this option generates an embedding where the local densities are encouraged to be
correlated with those in the original space.
Strength of local density preservation.
Controls the regularization weight of the density correlation term in densMAP. Higher values
prioritize density preservation over the UMAP objective, and vice versa for values closer to zero.
Setting this parameter to zero is equivalent to running the original UMAP algorithm.
Drop duplicate rows before embedding.
If you have more duplicates than you have `n_neighbors` you can have the identical data points lying
in different regions of your space. It also violates the definition of a metric. This option will
remove duplicates before embedding, and then map the original data points back to the reduced space. Duplicate
data points will be placed in the exact same location as the original data points.
A random number to initialize the algorithm for reproducibility.
Maintain links for n nearest neighbours only in graph.
Values must be in the following range:
```javascript theme={null}
1 ≤ links_top_n ≤ 15
```
Scaling factor for the coordinates.
The maximum (normalized) coordinates in positive and negative X and Y directions. Acts like a zoom, with
a scale of 1 corresponding to zooming out to the maximum (maximal space between nodes), and 0 to the densest
layout.
If set to `"auto"`, will try to determine an appropriate scale taking into account the number of nodes.
If set to `null`, only changes calculated coordinates to ensure they're within the allowed limits (16.000).
null.
string.
number.
Values must be in the following range:
```javascript theme={null}
0 < {_} ≤ 1
```
# vectorize_dataset
Source: https://docs.graphext.com/api-docs/analyse/embed/vectorize_dataset
Create a vectorized (numeric) dataset, (optionally) of reduced dimensionality.
Many machine learning and AI algorithms expect their input data to be in pure numerical form, i.e. not containing
categorical variables, missing values etc. This step converts arbitrary datasets, potentially containing non-numerical
variables and NaNs, into this expected form. It does this by defining for each possible type of input column a transformation
from non-numeric to numeric values. As an example, ordered categorical variables (ordinals) such as the day of week,
may be converted into a series of numbers (0..7). Non-ordered categorical variables of low-cardinality (containing few
different categories) may be expanded into multiple new columns of 0s and 1s, indicating whether each row belongs to a
specific category or not. Similar transformations are applied to dates, multivalued categoricals etc.
NaNs are imputed (replaced) with an appropriate value from the corresponding column (e.g. the median in a quantitative
column). In addition, a new column of 0s and 1s is added, indicating whether the original column had a missing value or not.
The resulting dataset will almost certainly not contain the same number of columns as the original (as the example of
categorical variables shows), and for simplicity, its columns will simply be numbered.
If desired, the `n_components` parameter may be used to select how many columns the new dataset should have, and if
this is smaller than would result normally, a dimensionality reduction will be applied ([UMAP](https://umap-learn.readthedocs.io/en/latest/)
by default).
The resulting numerical representation of the original data points aims to preserve the structure of similarities.
I.e. if two original rows are similar to each other, than their (potentially reduced) numerical representations should
also be similar. Equally, two very different rows should have representations that are also very different.
Note, if you need the output as a column of embedding vectors, rather than a dataset, use [`embed_dataset`](../embed_dataset/)
instead.
## Usage
The following examples show how the step can be used in a recipe.
The following, simplest, example, creates a new dataset containing a (potentially different) number of only numeric columns, where each row corresponds to its original row, and hopefully capturing the same or most of its information.
```stan theme={null}
vectorize_dataset(ds) -> (ds_vec)
```
The following example will convert and reduce the input dataset to a purely numeric dataset of 10 columns. After normalization, the `date` column will be multiplied by 0.5 to reduces its weight relative to the others. The column `age` on the other hand will be given more importance. Also, 15 neighbours are considered for each data point in UMAP, so that we give more importance to the similarity between nearby points and less importance to the global structure of the data when calculating the numeric representation of the dataset.
```stan theme={null}
vectorize_dataset(ds, {
"n_components": 10,
"weights": {"date": 0.5, "age": 2},
"n_neighbours": 15,
}) -> (ds_vec)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
vectorize_dataset(ds: dataset, {
"param": value,
...
}) -> (ds_out: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An arbitrary input dataset.
A new dataset containing only quantitative columns without missing values.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Algorithm.
The name of a supported dimensionality reduction algorithm.
Values must be one of the following:
* `umap`
Toggle encoding of feature columns.
When enabled, Graphext will auto-convert any column types to the numeric type before
(optionally) reducing the data's dimensionality. How this conversion is done can be
configured using the `feature_encoder` option below.
If disabled, the dimensionality reduction algorithm applied in this step will
assume that input data is already numerical and doesn't contain any missing
values.
Configures encoding of feature columns.
By default (`null`), Graphext chooses automatically how to convert any column types the model
may not understand natively to a numeric type.
A configuration object can be passed instead to overwrite specific parameter values with respect
to their default values.
Numeric encoder.
Configures encoding of numeric features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Further parameters passed to the `scaler` function.
Details depend no the particular scaler used.
Boolean encoder.
Configures encoding of boolean features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Ordinal encoder.
Configures encoding of categorical features that have a natural order.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Category encoder.
May contain either a single configuration for all categorical variables, or two different configurations
for low- and high-cardinality variables. For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for categories with fewer than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Multilabel encoder.
Configures encoding of multivalued categorical features (variable length lists of categories,
or the semantic type `list[category]` for short). May contain either a single configuration for
all multilabel variables, or two different configurations for low- and high-cardinality variables.
For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for mulitabel columns with fewer than `cardinality_threshold` unique categories/labels.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Datetime encoder.
Configures encoding of datetime (timestamp) features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
A list of numerical components to extract.
Will create one numeric column for each component.
Each item in array.
Values must be one of the following:
`day` `dayofweek` `dayofyear` `hour` `minute` `month` `quarter` `season` `second` `week` `weekday` `weekofyear` `year`
A list of cyclical time features to extract.
"Cycles" are numerical transformations of features that should be represented on a circle. E.g. months,
ranging from 1 to 12, should be arranged such that 12 and 1 are next to each other, rather than on
opposite ends of a linear scale. We represent such cyclical time features on a circle by creating two
columns for each original feature: the sin and cos of the numerical feature after appropriate scaling.
Each item in array.
Values must be one of the following:
* `day`
* `dayofweek`
* `dayofyear`
* `hour`
* `month`
Whether to include the epoch as new feature (seconds since 01/01/1970).
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Embedding/vector encoder.
Configures encoding of multivalued numerical features (variable length lists of numbers, i.e. vectors, or the semantic type `list[number]` for short).
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Text encoder.
Configures encoding of text (natural language) features. Currently only allows
[Tf-Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) embeddings to represent texts. If you wish
to use other embeddings, e.g. semantic, Word2Vec etc., transform your text column first using
another step, and then use that result instead of the original texts.
Texts are *excluded* by default from the overall encoding of the dataset. See parameter
`include_text_features` below to active it.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Parameters to be passed to the text encoder (Tf-Idf parameters only for now).
See [scikit-learn's documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)
for detailed parameters and their explanation.
How many output features to generate.
The resulting Tf-Idf vectors will be reduced to these many dimensions (columns) using scikit-learn's
[truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
This performs a kind of [latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
By default we will reduce to 200 components.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_components ≤ 1024
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Whether to include or ignore text columns during the processing of input data.
Enabling this will convert texts to their Tf-Idf representation. Each text will be
converted to an N-dimensional vector in which each component measures the relative
"over-representation" of a specific word (or n-gram) relative to its overall
frequency in the whole dataset. This is disabled by default because it will
often be better to convert texts explicitly using a previous step, such as
`embed_text` or `embed_text_with_model`.
Weights used to multiply the normalized columns/features after vectorization.
Should be a dictionary/object of `{"column_name": weight, ...}` items. Will be scaled using the
parameters `weights_max`, and `weights_exp` before being applied. So only the relative weight of
the columns is important here, not their absolute values.
A `"column_name": numeric_weight` pair.
Each column name must refer to an existing column in the dataset.
* `{"date": 0.5, "age": 2}`
Weights used to multiply the normalized columns/features after vectorization.
Should be a dictionary/object of `"type": weight"` items. Will be scaled using the parameters
`weights_max`, and `weights_exp` before being applied. So only the relative weight of the columns
is important here, not their absolute values.
Weight for columns of type `Number`
Weight for columns of type `Datetime`
Weight for columns of type `Category`
Weight for columns of type `Ordinal`
Weight for columns of type `Embedding` (`List[Number]`).
Weight for columns of type `Multilabel` (`List[Category]`).
Maximum weight to scale the normalized columns with.
Values must be in the following range:
```javascript theme={null}
0 ≤ weights_max < inf
```
Weight exponent.
Weights will be raised to this power before(!) scaling to `weights_max`. This allows for a non-linear
mapping from input weights to those used eventually to multiply the normalized columns.
Number of neighbours.
Use smaller numbers to concentrate on the local structure in the data, and larger values to focus on the
more global structure.
For further details see [here](https://umap-learn.readthedocs.io/en/latest/parameters.html#n-neighbors).
Values must be in the following range:
```javascript theme={null}
1 ≤ n_neighbors < inf
```
Minimum distance between reduced data points.
Controls how tightly UMAP is allowed to pack points together in the reduced space. Smaller values will
lead to points more tightly packed together (potentially useful if result is used to cluster the points).
Larger values will distribute points with more space between them (which may be desirable for visualization,
or to focus more on the global structure of the date).
For further details see [here](https://umap-learn.readthedocs.io/en/latest/parameters.html#min-dist).
Values must be in the following range:
```javascript theme={null}
0 ≤ min_dist < inf
```
Dimensionality of the reduced data. Fixed at 2 for the purpose of a layout's x and y coordinates.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_components < inf
```
Metric to use for measuring similarity between data points.
Values must be one of the following:
`euclidean` `manhattan` `chebyshev` `minkowski` `canberra` `braycurtis` `haversine` `mahalanobis` `wminkowski` `seuclidean` `cosine` `correlation` `hamming` `jaccard` `dice` `russellrao` `kulsinski` `rogerstanimoto` `sokalmichener` `sokalsneath` `yule`
Number of training iterations used in optimizing the embedding.
Larger values result in more accurate embeddings. If `null` is specified a value will be
selected based on the size of the input dataset (200 for large datasets, 500 for small).
How to initialize the low dimensional embedding.
When "spectral", uses a (relatively expensive) spectral embedding. "pca" uses the first `n_components`
from a principal component analysis. "tswspectral" is a cheaper alternative to "spectral". When "random",
assigns initial embedding positions at random. This uses the least amount of memory and time but may make UMAP
slower to converge on the optimal embedding. "auto" selects between "spectral" and "random" automatically
depending on the size of the dataset.
Values must be one of the following:
* `spectral`
* `pca`
* `tswspectral`
* `random`
* `auto`
Avoid excessive memory use.
For some datasets nearest neighbor computations can consume a lot of memory. If you find
the step is failing due to memory constraints, consider setting this option to `true`.
This approach is more computationally expensive, but avoids excessive memory use. Setting
it to "auto", will enable this mode automatically depending on the size of the dataset.
Values must be one of the following:
* `True`
* `False`
* `auto`
* `None`
Target variable (labels) for supervised dimensionality reduction.
Name of the column that contains your target values (labels).
Weighting factor between features and target.
A value of 0.0 weights entirely on data, and a value of 1.0 weights entirely on target. The
default of 0.5 balances the weighting equally between data and target.
Try to better preserve local densities in the data.
Specifies whether the density-augmented objective of densMAP should be used for optimization.
Turning on this option generates an embedding where the local densities are encouraged to be
correlated with those in the original space.
Strength of local density preservation.
Controls the regularization weight of the density correlation term in densMAP. Higher values
prioritize density preservation over the UMAP objective, and vice versa for values closer to zero.
Setting this parameter to zero is equivalent to running the original UMAP algorithm.
Drop duplicate rows before embedding.
If you have more duplicates than you have `n_neighbors` you can have the identical data points lying
in different regions of your space. It also violates the definition of a metric. This option will
remove duplicates before embedding, and then map the original data points back to the reduced space. Duplicate
data points will be placed in the exact same location as the original data points.
A random number to initialize the algorithm for reproducibility.
Maintain links for n nearest neighbours only in graph.
Values must be in the following range:
```javascript theme={null}
1 ≤ links_top_n ≤ 15
```
Scaling factor for the coordinates.
The maximum (normalized) coordinates in positive and negative X and Y directions. Acts like a zoom, with
a scale of 1 corresponding to zooming out to the maximum (maximal space between nodes), and 0 to the densest
layout.
If set to `"auto"`, will try to determine an appropriate scale taking into account the number of nodes.
If set to `null`, only changes calculated coordinates to ensure they're within the allowed limits (16.000).
null.
string.
number.
Values must be in the following range:
```javascript theme={null}
0 < {_} ≤ 1
```
# association_rules
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/association_rules
Calculate association rules for a items/products in a dataset of transactions.
This is a form of market basket analysis. It analyses items (products) that occur unusually
frequent together in a set of transactions (baskets).
The step creates an association rule, such as A->B, between items A and B, if the presence of
A makes the presence of B in the same session N times more likely.
For further details about the algorithm see e.g.
[association rule learning](https://en.wikipedia.org/wiki/Association_rule_learning).
## Usage
The following example shows how the step can be used in a recipe.
The following call creates rules between pairs of items A and B, if:
* A occurs in at least 7 sessions
* B occurs in at least 25% of sessions containing A
* The presence of A in a session makes the presence of B in the same session at least twice as likely.
Note that the last condition is equivalent to saying that the overall frequency of B in all sessions must be less than 12.5% (half of 25%). In other words, a minimum lift of 2 means that the frequency of B, in sessions already containing A, must be twice the background frequency of B in general.
As an example, the percentage of shopping baskets containing milk (item B) may be 10%. However, amongst those baskets already containing cereals, the percentage containing milk is likely to be higher. If milk occured e.g. in 30% of baskets also having cereals, than the lift of the rule cereal->milk would be 3. The buying of cereal make the buying of milk 3 times more likely.
```stan theme={null}
association_rules(transactions, {
"item_id": "product_id",
"session_id": "order_id",
"item_label": "product_name",
"min_support": 7,
"min_confidence": 25,
"min_lift": 2
}) -> (rules)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
association_rules(transactions: dataset, {
"param": value,
...
}) -> (rules: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A long input dataset with one row per item (product) and session (basket). In other words, sessions
or baskets should be \_dis\_aggregated, but each row should uniquely identify the item/product *and*
session/basket by id or name.
A new output dataset containing products and rules, connected into a network such that products are
linked to the association rules in which they occur.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Name of column uniquely identifying all items/products.
Name(s) of column(s) uniquely identifying all sessions/baskets/orders.
Each item in array.
Column used to label items in a user-friendly manner.
Minimum size of itemsets to identify.
E.g. an itemsize of 3 means association rules will have 2 antecedents (e.g. A, B)
and 1 consequent (C), resulting in rules of the form (A, B) -> C. The step will
currently generate only single items as consequents.
Values must be in the following range:
```javascript theme={null}
2 ≤ itemset_min ≤ 5
```
Maximum size of itemsets to identify.
E.g. an itemsize of 3 means association rules will have 2 antecedents (e.g. A, B)
and 1 consequent (C), resulting in rules of the form (A, B) -> C. The step will
currently generate only single items as consequents.
Values must be in the following range:
```javascript theme={null}
2 ≤ itemset_max ≤ 5
```
Minimum Support.
Minimum support of a rule antecedent. If it is \< 1 it will be taken as a proportion.
In any other case it will be expected as a positive integer representing the count.
Create rule A->B only if A occurred in at least this many sessions.
number.
Values must be in the following range:
```javascript theme={null}
0 < {_} < 1
```
integer.
Values must be in the following range:
```javascript theme={null}
1 ≤ {_} < inf
```
Minimum Confidence.
Expressed as a percentage. Include link A->B only if B occurred in at least this
percentage of sessions also containing A.
Values must be in the following range:
```javascript theme={null}
0 ≤ min_confidence ≤ 100
```
Minimum Lift.
Expressed as multipler/ratio. Include link A->B only if A makes the presence of B in the same
sessions at least this many times more likely.
Metric for link weight.
Which association rule metric to use as the weight of links in the network generated by this step.
Values must be one of the following:
`itemset_support_abs` `itemset_support_pct` `antecedent_support_abs` `antecedent_support_pct` `consequent_support_abs` `consequent_support_pct` `rule_confidence_pct` `rule_lift_abs` `rule_lift_pct`
Whether to link items to rules.
Otherwise, a product (antecedent) will be linked only to other products (consequent).
Only keep N links with largest weight.
This applies individually to each node in the network, filtering its outgoing links to keep only
the first N by weight. The value of weights itself is selected using the `weight_metric` parameter,
i.e. corresponds to one of the association rule metrics (support, confidence etc.). If `null`,
all links will be kept.
Definition of desired aggregations for (consequent) items.
A dictionary mapping original columns to new aggregated columns, specifying an aggregation function for each.
*Aggregations* are functions that reduce all the values in a particular column of a single item/product
to a single summary value for that item/product. E.g. a `sum` aggregation of column A calculates a single
total by adding up all the values in A belonging to each item.
Possible aggregations functions accepted as `func` parameters are:
* `n`, `size` or `count`: calculate number of rows in group
* `sum`: sum total of values
* `mean`: take mean of values
* `max`: take max of values
* `min`: take min of values
* `first`: take first item found
* `last`: take last item found
* `unique`: collect a list of unique values
* `n_unique`: count the number of unique values
* `list`: collect a list of all values
* `concatenate`: convert all values to text and concatenate them into one long text
* `concat_lists`: concatenate lists in all rows into a single larger list
* `count_where`: number of rows in which the column matches a value, needs parameter `value` with the value that you want to count
* `percent_where`: percentage of the column where the column matches a value, needs parameter `value` with the value that you want to count
Note that in the case of `count_where` and `percent_where` an additional `value` parameter is required.
Definition of desired aggregations for rules (all items in rule).
A dictionary mapping original columns to new aggregated columns, specifying an aggregation function for each.
*Aggregations* are functions that reduce all the values in a particular column of a single item/product
to a single summary value for that item/product. E.g. a `sum` aggregation of column A calculates a single
total by adding up all the values in A belonging to each item.
Possible aggregations functions accepted as `func` parameters are:
* `n`, `size` or `count`: calculate number of rows in group
* `sum`: sum total of values
* `mean`: take mean of values
* `max`: take max of values
* `min`: take min of values
* `first`: take first item found
* `last`: take last item found
* `unique`: collect a list of unique values
* `n_unique`: count the number of unique values
* `list`: collect a list of all values
* `concatenate`: convert all values to text and concatenate them into one long text
* `concat_lists`: concatenate lists in all rows into a single larger list
* `count_where`: number of rows in which the column matches a value, needs parameter `value` with the value that you want to count
* `percent_where`: percentage of the column where the column matches a value, needs parameter `value` with the value that you want to count
Note that in the case of `count_where` and `percent_where` an additional `value` parameter is required.
# cluster_dataset
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/cluster_dataset
Identify clusters in the dataset.
Applies a clustering algorithm after vectorizing the input dataset (converting its columns to numeric-only and no
missing data), and optionally reducing its dimensionality.
Essentially applies the separate step `vectorize_dataset`, followed by a clustering algorithm
([HDBSCAN](https://hdbscan.readthedocs.io/en/latest/index.html) by default). The result is a column of cluster IDs.
For further detail on HDBSCAN's parameters see its documentation [here (for usage)](https://hdbscan.readthedocs.io/en/latest/parameter_selection.html#)
and [here (for its API)](https://hdbscan.readthedocs.io/en/latest/api.html).
## Usage
The following example shows how the step can be used in a recipe.
The following configuration applies clustering with the default values:
```stan theme={null}
cluster_dataset(ds, {
"algorithm": "hdbscan",
"min_cluster_size": 120,
"min_samples": 15,
"reduce": {
"weights": null,
"weights_max": 32,
"weights_exp": 2,
"algorithm": "umap",
"n_components": 10,
"n_neighbors": 100,
"min_dist": 0,
"random_state": 42
}
}) -> (ds.cluster)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
cluster_dataset(ds: dataset, {
"param": value,
...
}) -> (*cluster: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An arbitrary input dataset.
One or two columns containing the clustering results. If one column name is provided, the single output
will contain the cluster labels. If two column names are provided, the second column will contain the
the probability that a data points belongs to the assigned cluster.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
The metric used to calculate similarity between data points.
Values must be one of the following:
`euclidean` `manhattan` `chebyshev` `minkowski` `canberra` `braycurtis` `haversine` `mahalanobis` `wminkowski` `seuclidean` `cosine` `correlation` `hamming` `jaccard` `dice` `russellrao` `kulsinski` `rogerstanimoto` `sokalmichener` `sokalsneath` `yule`
Algorithm to use.
The name of a supported clustering algorithm (currently allows `"hdbscan"` only).
Values must be one of the following:
* `hdbscan`
Minimum cluster size.
The minimum size for considering a region of dense data points a proper cluster.
Values must be in the following range:
```javascript theme={null}
1 ≤ min_cluster_size < inf
```
The larger the value, the more conservative the clustering.
More points will be declared as noise, and clusters will be restricted to progressively more dense areas.
Values must be in the following range:
```javascript theme={null}
1 ≤ min_samples < inf
```
Umap configuration. See more [here](https://umap-learn.readthedocs.io/en/latest/parameters.html).
Params for dimensionality reduction.
Algorithm.
The name of a supported dimensionality reduction algorithm.
Values must be one of the following:
* `umap`
Toggle encoding of feature columns.
When enabled, Graphext will auto-convert any column types to the numeric type before
(optionally) reducing the data's dimensionality. How this conversion is done can be
configured using the `feature_encoder` option below.
If disabled, the dimensionality reduction algorithm applied in this step will
assume that input data is already numerical and doesn't contain any missing
values.
Configures encoding of feature columns.
By default (`null`), Graphext chooses automatically how to convert any column types the model
may not understand natively to a numeric type.
A configuration object can be passed instead to overwrite specific parameter values with respect
to their default values.
Numeric encoder.
Configures encoding of numeric features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Further parameters passed to the `scaler` function.
Details depend no the particular scaler used.
Boolean encoder.
Configures encoding of boolean features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Ordinal encoder.
Configures encoding of categorical features that have a natural order.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Category encoder.
May contain either a single configuration for all categorical variables, or two different configurations
for low- and high-cardinality variables. For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for categories with fewer than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Multilabel encoder.
Configures encoding of multivalued categorical features (variable length lists of categories,
or the semantic type `list[category]` for short). May contain either a single configuration for
all multilabel variables, or two different configurations for low- and high-cardinality variables.
For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for mulitabel columns with fewer than `cardinality_threshold` unique categories/labels.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Datetime encoder.
Configures encoding of datetime (timestamp) features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
A list of numerical components to extract.
Will create one numeric column for each component.
Each item in array.
Values must be one of the following:
`day` `dayofweek` `dayofyear` `hour` `minute` `month` `quarter` `season` `second` `week` `weekday` `weekofyear` `year`
A list of cyclical time features to extract.
"Cycles" are numerical transformations of features that should be represented on a circle. E.g. months,
ranging from 1 to 12, should be arranged such that 12 and 1 are next to each other, rather than on
opposite ends of a linear scale. We represent such cyclical time features on a circle by creating two
columns for each original feature: the sin and cos of the numerical feature after appropriate scaling.
Each item in array.
Values must be one of the following:
* `day`
* `dayofweek`
* `dayofyear`
* `hour`
* `month`
Whether to include the epoch as new feature (seconds since 01/01/1970).
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Embedding/vector encoder.
Configures encoding of multivalued numerical features (variable length lists of numbers, i.e. vectors, or the semantic type `list[number]` for short).
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Text encoder.
Configures encoding of text (natural language) features. Currently only allows
[Tf-Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) embeddings to represent texts. If you wish
to use other embeddings, e.g. semantic, Word2Vec etc., transform your text column first using
another step, and then use that result instead of the original texts.
Texts are *excluded* by default from the overall encoding of the dataset. See parameter
`include_text_features` below to active it.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Parameters to be passed to the text encoder (Tf-Idf parameters only for now).
See [scikit-learn's documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)
for detailed parameters and their explanation.
How many output features to generate.
The resulting Tf-Idf vectors will be reduced to these many dimensions (columns) using scikit-learn's
[truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
This performs a kind of [latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
By default we will reduce to 200 components.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_components ≤ 1024
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Whether to include or ignore text columns during the processing of input data.
Enabling this will convert texts to their Tf-Idf representation. Each text will be
converted to an N-dimensional vector in which each component measures the relative
"over-representation" of a specific word (or n-gram) relative to its overall
frequency in the whole dataset. This is disabled by default because it will
often be better to convert texts explicitly using a previous step, such as
`embed_text` or `embed_text_with_model`.
Weights used to multiply the normalized columns/features after vectorization.
Should be a dictionary/object of `{"column_name": weight, ...}` items. Will be scaled using the
parameters `weights_max`, and `weights_exp` before being applied. So only the relative weight of
the columns is important here, not their absolute values.
A `"column_name": numeric_weight` pair.
Each column name must refer to an existing column in the dataset.
* `{"date": 0.5, "age": 2}`
Weights used to multiply the normalized columns/features after vectorization.
Should be a dictionary/object of `"type": weight"` items. Will be scaled using the parameters
`weights_max`, and `weights_exp` before being applied. So only the relative weight of the columns
is important here, not their absolute values.
Weight for columns of type `Number`
Weight for columns of type `Datetime`
Weight for columns of type `Category`
Weight for columns of type `Ordinal`
Weight for columns of type `Embedding` (`List[Number]`).
Weight for columns of type `Multilabel` (`List[Category]`).
Maximum weight to scale the normalized columns with.
Values must be in the following range:
```javascript theme={null}
0 ≤ weights_max < inf
```
Weight exponent.
Weights will be raised to this power before(!) scaling to `weights_max`. This allows for a non-linear
mapping from input weights to those used eventually to multiply the normalized columns.
Number of neighbours.
Use smaller numbers to concentrate on the local structure in the data, and larger values to focus on the
more global structure.
For further details see [here](https://umap-learn.readthedocs.io/en/latest/parameters.html#n-neighbors).
Values must be in the following range:
```javascript theme={null}
1 ≤ n_neighbors < inf
```
Minimum distance between reduced data points.
Controls how tightly UMAP is allowed to pack points together in the reduced space. Smaller values will
lead to points more tightly packed together (potentially useful if result is used to cluster the points).
Larger values will distribute points with more space between them (which may be desirable for visualization,
or to focus more on the global structure of the date).
For further details see [here](https://umap-learn.readthedocs.io/en/latest/parameters.html#min-dist).
Values must be in the following range:
```javascript theme={null}
0 ≤ min_dist < inf
```
Dimensionality of the reduced data.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_components < inf
```
Metric to use for measuring similarity between data points.
Values must be one of the following:
`euclidean` `manhattan` `chebyshev` `minkowski` `canberra` `braycurtis` `haversine` `mahalanobis` `wminkowski` `seuclidean` `cosine` `correlation` `hamming` `jaccard` `dice` `russellrao` `kulsinski` `rogerstanimoto` `sokalmichener` `sokalsneath` `yule`
Number of training iterations used in optimizing the embedding.
Larger values result in more accurate embeddings. If `null` is specified a value will be
selected based on the size of the input dataset (200 for large datasets, 500 for small).
How to initialize the low dimensional embedding.
When "spectral", uses a (relatively expensive) spectral embedding. "pca" uses the first `n_components`
from a principal component analysis. "tswspectral" is a cheaper alternative to "spectral". When "random",
assigns initial embedding positions at random. This uses the least amount of memory and time but may make UMAP
slower to converge on the optimal embedding. "auto" selects between "spectral" and "random" automatically
depending on the size of the dataset.
Values must be one of the following:
* `spectral`
* `pca`
* `tswspectral`
* `random`
* `auto`
Avoid excessive memory use.
For some datasets nearest neighbor computations can consume a lot of memory. If you find
the step is failing due to memory constraints, consider setting this option to `true`.
This approach is more computationally expensive, but avoids excessive memory use. Setting
it to "auto", will enable this mode automatically depending on the size of the dataset.
Values must be one of the following:
* `True`
* `False`
* `auto`
* `None`
Target variable (labels) for supervised dimensionality reduction.
Name of the column that contains your target values (labels).
Weighting factor between features and target.
A value of 0.0 weights entirely on data, and a value of 1.0 weights entirely on target. The
default of 0.5 balances the weighting equally between data and target.
Try to better preserve local densities in the data.
Specifies whether the density-augmented objective of densMAP should be used for optimization.
Turning on this option generates an embedding where the local densities are encouraged to be
correlated with those in the original space.
Strength of local density preservation.
Controls the regularization weight of the density correlation term in densMAP. Higher values
prioritize density preservation over the UMAP objective, and vice versa for values closer to zero.
Setting this parameter to zero is equivalent to running the original UMAP algorithm.
Drop duplicate rows before embedding.
If you have more duplicates than you have `n_neighbors` you can have the identical data points lying
in different regions of your space. It also violates the definition of a metric. This option will
remove duplicates before embedding, and then map the original data points back to the reduced space. Duplicate
data points will be placed in the exact same location as the original data points.
A random number to initialize the algorithm for reproducibility.
# cluster_embeddings
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/cluster_embeddings
Identify clusters using the distance between provided embeddings.
Eqivalent to `cluster_dataset`, but instead of a dataset expects a column of embeddings as input. The input may e.g.
be word2vec embeddings from an `embed_text` step, or whole dataset embeddings from an `embed_dataset` step.
Optionally reduces the dimensionality of the embeddings (by default using [UMAP](https://umap-learn.readthedocs.io/en/latest/)).
This may help with making the data denser (counteracting the ["curse-of-dimensionality"](https://en.wikipedia.org/wiki/Curse_of_dimensionality)),
and thus making it potentially easier to identify clusters.
The clustering algorithm used by default is ([HDBSCAN](https://hdbscan.readthedocs.io/en/latest/index.html)), which
produces a column of positive cluster IDs, or -1 if a data point is considered noise (not belonging to any cluster).
For further detail on HDBSCAN's parameters see its documentation [here (for usage)](https://hdbscan.readthedocs.io/en/latest/parameter_selection.html#)
and [here (for its API)](https://hdbscan.readthedocs.io/en/latest/api.html).
## Usage
The following example shows how the step can be used in a recipe.
The following configuration applies clustering with the default values:
```stan theme={null}
cluster_embeddings(ds, {
"algorithm": "hdbscan",
"min_cluster_size": 120,
"min_samples": 15,
"reduce": {
"weights": null,
"weights_max": 32,
"weights_exp": 2,
"algorithm": "umap",
"n_components": 10,
"n_neighbors": 100,
"min_dist": 0,
"random_state": 42
}
}) -> (ds.cluster)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
cluster_embeddings(embeddings: list[number], {
"param": value,
...
}) -> (*cluster: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column of embeddings (list/vectors of numbers).
One or two columns containing the clustering results. If one column name is provided, the single output
will contain the cluster labels. If two column names are provided, the second column will contain the
the probability that a data points belongs to the assigned cluster.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
The metric used to calculate similarity between data points.
Values must be one of the following:
`euclidean` `manhattan` `chebyshev` `minkowski` `canberra` `braycurtis` `haversine` `mahalanobis` `wminkowski` `seuclidean` `cosine` `correlation` `hamming` `jaccard` `dice` `russellrao` `kulsinski` `rogerstanimoto` `sokalmichener` `sokalsneath` `yule`
Algorithm to use.
The name of a supported clustering algorithm (currently allows `"hdbscan"` only).
Values must be one of the following:
* `hdbscan`
Minimum cluster size.
The minimum size for considering a region of dense data points a proper cluster.
Values must be in the following range:
```javascript theme={null}
1 ≤ min_cluster_size < inf
```
The larger the value, the more conservative the clustering.
More points will be declared as noise, and clusters will be restricted to progressively more dense areas.
Values must be in the following range:
```javascript theme={null}
1 ≤ min_samples < inf
```
Umap configuration. See more [here](https://umap-learn.readthedocs.io/en/latest/parameters.html).
Params for dimensionality reduction.
Algorithm.
The name of a supported dimensionality reduction algorithm.
Values must be one of the following:
* `umap`
Toggle encoding of feature columns.
When enabled, Graphext will auto-convert any column types to the numeric type before
(optionally) reducing the data's dimensionality. How this conversion is done can be
configured using the `feature_encoder` option below.
If disabled, the dimensionality reduction algorithm applied in this step will
assume that input data is already numerical and doesn't contain any missing
values.
Configures encoding of feature columns.
By default (`null`), Graphext chooses automatically how to convert any column types the model
may not understand natively to a numeric type.
A configuration object can be passed instead to overwrite specific parameter values with respect
to their default values.
Numeric encoder.
Configures encoding of numeric features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Further parameters passed to the `scaler` function.
Details depend no the particular scaler used.
Boolean encoder.
Configures encoding of boolean features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Ordinal encoder.
Configures encoding of categorical features that have a natural order.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Category encoder.
May contain either a single configuration for all categorical variables, or two different configurations
for low- and high-cardinality variables. For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for categories with fewer than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Multilabel encoder.
Configures encoding of multivalued categorical features (variable length lists of categories,
or the semantic type `list[category]` for short). May contain either a single configuration for
all multilabel variables, or two different configurations for low- and high-cardinality variables.
For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for mulitabel columns with fewer than `cardinality_threshold` unique categories/labels.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Datetime encoder.
Configures encoding of datetime (timestamp) features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
A list of numerical components to extract.
Will create one numeric column for each component.
Each item in array.
Values must be one of the following:
`day` `dayofweek` `dayofyear` `hour` `minute` `month` `quarter` `season` `second` `week` `weekday` `weekofyear` `year`
A list of cyclical time features to extract.
"Cycles" are numerical transformations of features that should be represented on a circle. E.g. months,
ranging from 1 to 12, should be arranged such that 12 and 1 are next to each other, rather than on
opposite ends of a linear scale. We represent such cyclical time features on a circle by creating two
columns for each original feature: the sin and cos of the numerical feature after appropriate scaling.
Each item in array.
Values must be one of the following:
* `day`
* `dayofweek`
* `dayofyear`
* `hour`
* `month`
Whether to include the epoch as new feature (seconds since 01/01/1970).
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Embedding/vector encoder.
Configures encoding of multivalued numerical features (variable length lists of numbers, i.e. vectors, or the semantic type `list[number]` for short).
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Text encoder.
Configures encoding of text (natural language) features. Currently only allows
[Tf-Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) embeddings to represent texts. If you wish
to use other embeddings, e.g. semantic, Word2Vec etc., transform your text column first using
another step, and then use that result instead of the original texts.
Texts are *excluded* by default from the overall encoding of the dataset. See parameter
`include_text_features` below to active it.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Parameters to be passed to the text encoder (Tf-Idf parameters only for now).
See [scikit-learn's documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)
for detailed parameters and their explanation.
How many output features to generate.
The resulting Tf-Idf vectors will be reduced to these many dimensions (columns) using scikit-learn's
[truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
This performs a kind of [latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
By default we will reduce to 200 components.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_components ≤ 1024
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Whether to include or ignore text columns during the processing of input data.
Enabling this will convert texts to their Tf-Idf representation. Each text will be
converted to an N-dimensional vector in which each component measures the relative
"over-representation" of a specific word (or n-gram) relative to its overall
frequency in the whole dataset. This is disabled by default because it will
often be better to convert texts explicitly using a previous step, such as
`embed_text` or `embed_text_with_model`.
Weights used to multiply the normalized columns/features after vectorization.
Should be a dictionary/object of `{"column_name": weight, ...}` items. Will be scaled using the
parameters `weights_max`, and `weights_exp` before being applied. So only the relative weight of
the columns is important here, not their absolute values.
A `"column_name": numeric_weight` pair.
Each column name must refer to an existing column in the dataset.
* `{"date": 0.5, "age": 2}`
Weights used to multiply the normalized columns/features after vectorization.
Should be a dictionary/object of `"type": weight"` items. Will be scaled using the parameters
`weights_max`, and `weights_exp` before being applied. So only the relative weight of the columns
is important here, not their absolute values.
Weight for columns of type `Number`
Weight for columns of type `Datetime`
Weight for columns of type `Category`
Weight for columns of type `Ordinal`
Weight for columns of type `Embedding` (`List[Number]`).
Weight for columns of type `Multilabel` (`List[Category]`).
Maximum weight to scale the normalized columns with.
Values must be in the following range:
```javascript theme={null}
0 ≤ weights_max < inf
```
Weight exponent.
Weights will be raised to this power before(!) scaling to `weights_max`. This allows for a non-linear
mapping from input weights to those used eventually to multiply the normalized columns.
Number of neighbours.
Use smaller numbers to concentrate on the local structure in the data, and larger values to focus on the
more global structure.
For further details see [here](https://umap-learn.readthedocs.io/en/latest/parameters.html#n-neighbors).
Values must be in the following range:
```javascript theme={null}
1 ≤ n_neighbors < inf
```
Minimum distance between reduced data points.
Controls how tightly UMAP is allowed to pack points together in the reduced space. Smaller values will
lead to points more tightly packed together (potentially useful if result is used to cluster the points).
Larger values will distribute points with more space between them (which may be desirable for visualization,
or to focus more on the global structure of the date).
For further details see [here](https://umap-learn.readthedocs.io/en/latest/parameters.html#min-dist).
Values must be in the following range:
```javascript theme={null}
0 ≤ min_dist < inf
```
Dimensionality of the reduced data.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_components < inf
```
Metric to use for measuring similarity between data points.
Values must be one of the following:
`euclidean` `manhattan` `chebyshev` `minkowski` `canberra` `braycurtis` `haversine` `mahalanobis` `wminkowski` `seuclidean` `cosine` `correlation` `hamming` `jaccard` `dice` `russellrao` `kulsinski` `rogerstanimoto` `sokalmichener` `sokalsneath` `yule`
Number of training iterations used in optimizing the embedding.
Larger values result in more accurate embeddings. If `null` is specified a value will be
selected based on the size of the input dataset (200 for large datasets, 500 for small).
How to initialize the low dimensional embedding.
When "spectral", uses a (relatively expensive) spectral embedding. "pca" uses the first `n_components`
from a principal component analysis. "tswspectral" is a cheaper alternative to "spectral". When "random",
assigns initial embedding positions at random. This uses the least amount of memory and time but may make UMAP
slower to converge on the optimal embedding. "auto" selects between "spectral" and "random" automatically
depending on the size of the dataset.
Values must be one of the following:
* `spectral`
* `pca`
* `tswspectral`
* `random`
* `auto`
Avoid excessive memory use.
For some datasets nearest neighbor computations can consume a lot of memory. If you find
the step is failing due to memory constraints, consider setting this option to `true`.
This approach is more computationally expensive, but avoids excessive memory use. Setting
it to "auto", will enable this mode automatically depending on the size of the dataset.
Values must be one of the following:
* `True`
* `False`
* `auto`
* `None`
Target variable (labels) for supervised dimensionality reduction.
Name of the column that contains your target values (labels).
Weighting factor between features and target.
A value of 0.0 weights entirely on data, and a value of 1.0 weights entirely on target. The
default of 0.5 balances the weighting equally between data and target.
Try to better preserve local densities in the data.
Specifies whether the density-augmented objective of densMAP should be used for optimization.
Turning on this option generates an embedding where the local densities are encouraged to be
correlated with those in the original space.
Strength of local density preservation.
Controls the regularization weight of the density correlation term in densMAP. Higher values
prioritize density preservation over the UMAP objective, and vice versa for values closer to zero.
Setting this parameter to zero is equivalent to running the original UMAP algorithm.
Drop duplicate rows before embedding.
If you have more duplicates than you have `n_neighbors` you can have the identical data points lying
in different regions of your space. It also violates the definition of a metric. This option will
remove duplicates before embedding, and then map the original data points back to the reduced space. Duplicate
data points will be placed in the exact same location as the original data points.
A random number to initialize the algorithm for reproducibility.
# cluster_network
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/cluster_network
Identify clusters in the network.
At the moment the only supported clustering algorithm is [Louvain](https://en.wikipedia.org/wiki/Louvain_modularity).
Louvain tries to identify the communities in a network by optimizing the modularity of the whole network, that is a
measure of the density of edges inside communities to edges outside communities. The result is a column of cluster IDs (integers),
where the value -1 is reserved for nodes in very small clusters, which are grouped into a "noise" cluster.
## Usage
The following example shows how the step can be used in a recipe.
The following configuration allows for smallish clusters and considers fewish data points as noise:
```stan theme={null}
cluster_network(ds.targets, ds.weights, {
"resolution": 0.3,
"noise": 5
}) -> (ds.cluster)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
cluster_network(targets: list[number], *weights: list[number], {
"param": value,
...
}) -> (cluster: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column containing link targets. Source is implied in the index.
A column containing cluster tags.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Clustering algorithm to use.
Only Louvain is currently supported.
Values must be one of the following:
* `louvain`
The higher this value the bigger the clusters.
Values must be in the following range:
```javascript theme={null}
0 < resolution ≤ 1
```
The larger the value, the more conservative the clustering.
Cluster with this number of nodes or less will be considered noise.
Values must be in the following range:
```javascript theme={null}
0 ≤ noise < inf
```
The *graphext advanced query syntax* used to select rows.
# cluster_subnetwork
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/cluster_subnetwork
Identify clusters in the network by filtering the input dataset.
At the moment the only supported clustering algorithm is [Louvain](https://en.wikipedia.org/wiki/Louvain_modularity).
Louvain tries to identify the communities in a network by optimizing the modularity of the whole network, that is a
measure of the density of edges inside communities to edges outside communities. The result is a column of cluster IDs (integers),
where the value -1 is reserved for nodes in very small clusters, which are grouped into a "noise" cluster.
## Usage
The following example shows how the step can be used in a recipe.
The following configuration allows for smallish clusters and considers fewish data points as noise:
```stan theme={null}
cluster_subnetwork(ds, {
"targets": "targets",
"weights": "weights",
"resolution": 0.3,
"noise": 5
}) -> (ds.cluster)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
cluster_subnetwork(ds_in: dataset, {
"param": value,
...
}) -> (cluster: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An input dataset to use as source of the network.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Name of column containing the link targets. Source is implied in the index.
Name of column containing the link weights.
The *graphext advanced query syntax* used to select rows.
Clustering algorithm to use.
Only Louvain is currently supported.
Values must be one of the following:
* `louvain`
The higher this value the bigger the clusters.
Values must be in the following range:
```javascript theme={null}
0 < resolution ≤ 1
```
The larger the value, the more conservative the clustering.
Cluster with this number of nodes or less will be considered noise.
Values must be in the following range:
```javascript theme={null}
0 ≤ noise < inf
```
# extract_node_betweenness
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/extract_node_betweenness
Calculate network node betweenness.
Calculates the [betweenness centrality](https://en.wikipedia.org/wiki/Betweenness_centrality)
for each node in the network. Betweenness centrality is a measure of the number of times a node acts
as a bridge along the shortest path between two other nodes.
## Usage
The following example shows how the step can be used in a recipe.
```stan theme={null}
extract_node_betweenness(ds.targets, ds.weights) -> (ds.betweenness)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
extract_node_betweenness(targets: list[number], *weights: list[number], {
"param": value,
...
}) -> (betweenness: number)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column containing link targets. Source is implied in the index.
An optional column containing link weights.
A column containing the betweenness metric for each node/row.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Whether the links are directed or not.
The maximum path length to consider when calculating the betweenness.
If cutoff is zero or negative then there is no such limit.
# extract_node_closeness
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/extract_node_closeness
Calculcate network node closeness.
Calculates the [closeness centrality](https://en.wikipedia.org/wiki/Closeness_centrality)
for each node in the network. Closeness centrality is a measure of how many steps are required
to access every other vertex from a given vertex. In other words, it finds the nodes best placed
to influence the entire network most quickly.
## Usage
The following example shows how the step can be used in a recipe.
```stan theme={null}
extract_node_closeness(ds.targets, ds.weights) -> (ds.closeness)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
extract_node_closeness(targets: list[number], *weights: list[number], {
"param": value,
...
}) -> (closeness: number)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column containing link targets. Source is implied in the index.
An optional column containing link weights.
Column containing how many steps is required to access every other vertex from a given vertex.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Which node connections to count.
Whether to
* `in`: count only a node's incoming links
* `out`: count only a node's outgoing links
* `all`/`both` count both incoming and outgoing links.
Values must be one of the following:
* `all`
* `out`
* `in`
* `both`
Whether to calculate the normalized closeness.
The maximum path length to consider when calculating the betweenness.
If cutoff is zero or negative then there is no such limit.
# extract_node_degree
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/extract_node_degree
Calculate network node degrees.
Calculates the [degree centrality](https://en.wikipedia.org/wiki/Degree_\(graph_theory\))
of each node in the network, i.e. the number of each node's incoming and/or outgoing connections.
## Usage
The following example shows how the step can be used in a recipe.
E.g. in a network of twitter accounts, where a directed link between nodes A and B indicates the number of times A has retweeted B, the following calculates the total number of retweets each account has received: Since in the example network links A→B and B→A can be different, we indicate that we want to interpret the network as *directed*. And since each link's weight is the number of retweets, we pass the retweets weight as the weights column. Keep in mind that both columns contain lists of numbers for each row.
```stan theme={null}
extract_node_degree(ds.targets, ds.retweets, {
"mode": "in",
"loops": false,
"directed": true
}) -> (ds.keywords)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
extract_node_degree(targets: list[number], *weights: list[number], {
"param": value,
...
}) -> (degree: number)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column containing link targets. Source is implied in the index.
An optional column containing link weights.
Column containing the number of incoming, outgoing or all connections for each row/node in the dataset.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Which node connections to count.
Whether to
* `in`: count only a node's incoming links
* `out`: count only a node's outgoing links
* `all`/`both` count both incoming and outgoing links.
Values must be one of the following:
* `all`
* `out`
* `in`
* `both`
Whether the links are directed or not.
Whether loops will be counted.
Loops are links of nodes to themselves.
# extract_node_pagerank
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/extract_node_pagerank
Calculate network node pagerank.
Calculates the [PageRank centrality](https://en.wikipedia.org/wiki/PageRank) for each node in the network.
PageRank is a measure of the importance of a node in a network. It is based on the idea that a node is important
if it is linked to by other important nodes. The algorithm is iterative and the importance of a node is calculated
as the sum of the importance of the nodes that link to it. The importance of a node is then distributed to the nodes
it links to. The algorithm is run until convergence. A damping factor is used to avoid the problem of dead ends.
For more information about the algorithm and its parameters see the [wikipedia entry](https://en.wikipedia.org/wiki/PageRank#Damping_factor)
or the original paper [here](http://infolab.stanford.edu/~backrub/google.html).
## Usage
The following example shows how the step can be used in a recipe.
```stan theme={null}
extract_node_pagerank(ds.targets, ds.weights) -> (ds.page_rank)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
extract_node_pagerank(targets: list[number], *weights: list[number], {
"param": value,
...
}) -> (pagerank: number)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column containing link targets. Source is implied in the index.
An optional column containing link weights.
Calculates the Google PageRank for the specified vertices.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Whether the links are directed or not.
The damping factor.
`1 - damping` is the PageRank value for nodes with no incoming links. It is also the probability of
resetting the random walk to a uniform distribution in each step.
Values must be in the following range:
```javascript theme={null}
0 ≤ damping ≤ 1
```
# Graph And Map
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/index
| Step | Fast | Description |
| -------------------------------------------------------------------------------------- | ---- | ----------------------------------------------------------------------------------------------------------------------- |
| [association\_rules](/api-docs/analyse/graph_and_map/association_rules) | | Calculate association rules for a items/products in a dataset of transactions |
| [cluster\_dataset](/api-docs/analyse/graph_and_map/cluster_dataset) | | Identify clusters in the dataset |
| [cluster\_embeddings](/api-docs/analyse/graph_and_map/cluster_embeddings) | | Identify clusters using the distance between provided embeddings |
| [cluster\_network](/api-docs/analyse/graph_and_map/cluster_network) | ⚡ | Identify clusters in the network |
| [cluster\_subnetwork](/api-docs/analyse/graph_and_map/cluster_subnetwork) | ⚡ | Identify clusters in the network by filtering the input dataset |
| [extract\_node\_betweenness](/api-docs/analyse/graph_and_map/extract_node_betweenness) | | Calculate network node betweenness |
| [extract\_node\_closeness](/api-docs/analyse/graph_and_map/extract_node_closeness) | | Calculcate network node closeness |
| [extract\_node\_degree](/api-docs/analyse/graph_and_map/extract_node_degree) | | Calculate network node degrees |
| [extract\_node\_pagerank](/api-docs/analyse/graph_and_map/extract_node_pagerank) | | Calculate network node pagerank |
| [layout\_coordinates](/api-docs/analyse/graph_and_map/layout_coordinates) | ⚡ | Create x, y positions for nodes from their geographical coordinates |
| [layout\_dataset](/api-docs/analyse/graph_and_map/layout_dataset) | | Reduce the dataset to 2 dimensions that can be mapped to x/y node positions |
| [layout\_igraph](/api-docs/analyse/graph_and_map/layout_igraph) | | Calculate layout, i.e. node positions, for a network |
| [layout\_network](/api-docs/analyse/graph_and_map/layout_network) | ⚡ | Compute a force-directed graph layout with a fast forceAtlas2 implementation |
| [layout\_treemap](/api-docs/analyse/graph_and_map/layout_treemap) | | Place nodes on the screen using a treemap layout |
| [link\_embeddings](/api-docs/analyse/graph_and_map/link_embeddings) | | Create network links between rows/nodes calculating the similarity of embeddings (vectors) |
| [link\_grouped\_embeddings](/api-docs/analyse/graph_and_map/link_grouped_embeddings) | | Create network links calculating the similarity of embeddings (vectors) within groups |
| [link\_rows](/api-docs/analyse/graph_and_map/link_rows) | | Create network links using explicit lists of target IDs, weights and other link attributes |
| [link\_rows\_by\_id](/api-docs/analyse/graph_and_map/link_rows_by_id) | | Create network links using one or more lists of target ids |
| [link\_rows\_by\_rownum](/api-docs/analyse/graph_and_map/link_rows_by_rownum) | | Create network links using explicit lists of target row numbers and optional weights |
| [link\_sequence\_items](/api-docs/analyse/graph_and_map/link_sequence_items) | | Create network links between consecutive pairs in a column of sequences |
| [link\_session\_items](/api-docs/analyse/graph_and_map/link_session_items) | | Link items (e.g. products) in sessions (baskets) if one item makes the presence of the other in the same session more … |
| [link\_similar\_columns](/api-docs/analyse/graph_and_map/link_similar_columns) | | Calculates all pair-wise column dependencies (by default mutual information) |
| [link\_similar\_rows](/api-docs/analyse/graph_and_map/link_similar_rows) | | Create network links calculating similarity between multidimensional and multitype documents |
| [merge\_links](/api-docs/analyse/graph_and_map/merge_links) | | Merge multiple sets of network link columns into a single unified link set |
# layout_coordinates
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/layout_coordinates
Create x, y positions for nodes from their geographical coordinates.
Used to create a Graph view in which rows/nodes are placed geographically, rather than according to
their (non-geographical) similarity.
## Usage
The following example shows how the step can be used in a recipe.
The following configuration transforms geographical coordinates into x and y points in the graph:
```stan theme={null}
layout_coordinates(ds.latitude, ds.longitude) -> (ds.x, ds.y)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
layout_coordinates(latitude: number, longitude: number, {
"param": value,
...
}) -> (x: column, y: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Numerical column with the latitude value.
Numerical column with the longitude value.
Numerical column with the x position in the graph.
Numerical column with the y position in the graph.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
A method to use to convert the coordinates to x, y positions.
Values must be one of the following:
* `webmercator`
* `spherical`
* `wgs84`
# layout_dataset
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/layout_dataset
Reduce the dataset to 2 dimensions that can be mapped to x/y node positions.
Based on the same vectorization and dimensionality reduction as the steps [`vectorize_dataset`](https://docs.graphext.com/api-docs/prepare/embed/vectorize_dataset/)
and [`embed_dataset`](https://docs.graphext.com/api-docs/prepare/embed/embed_dataset/). The only difference being that the number of dimensions (output columns) is
fixed to 2 (corresponding to x and y positions).
## Usage
The following examples show how the step can be used in a recipe.
The following, simplest, example, will convert the input dataset `ds` to purely numerical form, will reduce its dimensionality to just 2 using [UMAP](https://umap-learn.readthedocs.io/en/latest/), and will save those 2 dimensions in the columns `x` and `y`. The way that the x and y coordinates are calculated via dimensionality reduction should preserve the similarity between original rows. I.e., rows that are similar in the original dataset should have coordinates close to each other.
```stan theme={null}
layout_dataset(ds) -> (ds.x, ds.y)
```
The following example will numerically convert and reduce the input dataset to 2 dimensions: x and y. 15 neighbours are considered for each data point during dimensionality reduction (instead of the default 100), so that we give more importance to the similarity between nearby points and less importance to the global structure of the data when calculating the layout. Also, the minimum distance of neighbours is increased from 0.1 to 0.8, to create a less dense and overlapping layout.
```stan theme={null}
layout_dataset(ds, {
"algorithm": "umap",
"n_neighbors": 15,
"min_dist": 0.8,
"random_state": 42,
"scale": 200,
}) -> (ds.x, ds.y)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
layout_dataset(ds: dataset, {
"param": value,
...
}) -> (x: column, y: column, *links: column
)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An arbitrary input dataset.
Column containing the x coordinate for each row.
Column containing the y coordinate for each row.
Two optional columns holding the links between nearest neighbours in space of the created layout.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Algorithm.
The name of a supported dimensionality reduction algorithm.
Values must be one of the following:
* `umap`
Toggle encoding of feature columns.
When enabled, Graphext will auto-convert any column types to the numeric type before
(optionally) reducing the data's dimensionality. How this conversion is done can be
configured using the `feature_encoder` option below.
If disabled, the dimensionality reduction algorithm applied in this step will
assume that input data is already numerical and doesn't contain any missing
values.
Configures encoding of feature columns.
By default (`null`), Graphext chooses automatically how to convert any column types the model
may not understand natively to a numeric type.
A configuration object can be passed instead to overwrite specific parameter values with respect
to their default values.
Numeric encoder.
Configures encoding of numeric features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Further parameters passed to the `scaler` function.
Details depend no the particular scaler used.
Boolean encoder.
Configures encoding of boolean features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Ordinal encoder.
Configures encoding of categorical features that have a natural order.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Category encoder.
May contain either a single configuration for all categorical variables, or two different configurations
for low- and high-cardinality variables. For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for categories with fewer than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Multilabel encoder.
Configures encoding of multivalued categorical features (variable length lists of categories,
or the semantic type `list[category]` for short). May contain either a single configuration for
all multilabel variables, or two different configurations for low- and high-cardinality variables.
For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for mulitabel columns with fewer than `cardinality_threshold` unique categories/labels.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Datetime encoder.
Configures encoding of datetime (timestamp) features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
A list of numerical components to extract.
Will create one numeric column for each component.
Each item in array.
Values must be one of the following:
`day` `dayofweek` `dayofyear` `hour` `minute` `month` `quarter` `season` `second` `week` `weekday` `weekofyear` `year`
A list of cyclical time features to extract.
"Cycles" are numerical transformations of features that should be represented on a circle. E.g. months,
ranging from 1 to 12, should be arranged such that 12 and 1 are next to each other, rather than on
opposite ends of a linear scale. We represent such cyclical time features on a circle by creating two
columns for each original feature: the sin and cos of the numerical feature after appropriate scaling.
Each item in array.
Values must be one of the following:
* `day`
* `dayofweek`
* `dayofyear`
* `hour`
* `month`
Whether to include the epoch as new feature (seconds since 01/01/1970).
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Embedding/vector encoder.
Configures encoding of multivalued numerical features (variable length lists of numbers, i.e. vectors, or the semantic type `list[number]` for short).
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Text encoder.
Configures encoding of text (natural language) features. Currently only allows
[Tf-Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) embeddings to represent texts. If you wish
to use other embeddings, e.g. semantic, Word2Vec etc., transform your text column first using
another step, and then use that result instead of the original texts.
Texts are *excluded* by default from the overall encoding of the dataset. See parameter
`include_text_features` below to active it.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Parameters to be passed to the text encoder (Tf-Idf parameters only for now).
See [scikit-learn's documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)
for detailed parameters and their explanation.
How many output features to generate.
The resulting Tf-Idf vectors will be reduced to these many dimensions (columns) using scikit-learn's
[truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
This performs a kind of [latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
By default we will reduce to 200 components.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_components ≤ 1024
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Whether to include or ignore text columns during the processing of input data.
Enabling this will convert texts to their Tf-Idf representation. Each text will be
converted to an N-dimensional vector in which each component measures the relative
"over-representation" of a specific word (or n-gram) relative to its overall
frequency in the whole dataset. This is disabled by default because it will
often be better to convert texts explicitly using a previous step, such as
`embed_text` or `embed_text_with_model`.
Weights used to multiply the normalized columns/features after vectorization.
Should be a dictionary/object of `{"column_name": weight, ...}` items. Will be scaled using the
parameters `weights_max`, and `weights_exp` before being applied. So only the relative weight of
the columns is important here, not their absolute values.
A `"column_name": numeric_weight` pair.
Each column name must refer to an existing column in the dataset.
* `{"date": 0.5, "age": 2}`
Weights used to multiply the normalized columns/features after vectorization.
Should be a dictionary/object of `"type": weight"` items. Will be scaled using the parameters
`weights_max`, and `weights_exp` before being applied. So only the relative weight of the columns
is important here, not their absolute values.
Weight for columns of type `Number`
Weight for columns of type `Datetime`
Weight for columns of type `Category`
Weight for columns of type `Ordinal`
Weight for columns of type `Embedding` (`List[Number]`).
Weight for columns of type `Multilabel` (`List[Category]`).
Maximum weight to scale the normalized columns with.
Values must be in the following range:
```javascript theme={null}
0 ≤ weights_max < inf
```
Weight exponent.
Weights will be raised to this power before(!) scaling to `weights_max`. This allows for a non-linear
mapping from input weights to those used eventually to multiply the normalized columns.
Number of neighbours.
Use smaller numbers to concentrate on the local structure in the data, and larger values to focus on the
more global structure.
For further details see [here](https://umap-learn.readthedocs.io/en/latest/parameters.html#n-neighbors).
Values must be in the following range:
```javascript theme={null}
1 ≤ n_neighbors < inf
```
Minimum distance between reduced data points.
Controls how tightly UMAP is allowed to pack points together in the reduced space. Smaller values will
lead to points more tightly packed together (potentially useful if result is used to cluster the points).
Larger values will distribute points with more space between them (which may be desirable for visualization,
or to focus more on the global structure of the date).
For further details see [here](https://umap-learn.readthedocs.io/en/latest/parameters.html#min-dist).
Values must be in the following range:
```javascript theme={null}
0 ≤ min_dist < inf
```
Dimensionality of the reduced data. Fixed at 2 for the purpose of a layout's x and y coordinates.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_components < inf
```
Metric to use for measuring similarity between data points.
Values must be one of the following:
`euclidean` `manhattan` `chebyshev` `minkowski` `canberra` `braycurtis` `haversine` `mahalanobis` `wminkowski` `seuclidean` `cosine` `correlation` `hamming` `jaccard` `dice` `russellrao` `kulsinski` `rogerstanimoto` `sokalmichener` `sokalsneath` `yule`
Number of training iterations used in optimizing the embedding.
Larger values result in more accurate embeddings. If `null` is specified a value will be
selected based on the size of the input dataset (200 for large datasets, 500 for small).
How to initialize the low dimensional embedding.
When "spectral", uses a (relatively expensive) spectral embedding. "pca" uses the first `n_components`
from a principal component analysis. "tswspectral" is a cheaper alternative to "spectral". When "random",
assigns initial embedding positions at random. This uses the least amount of memory and time but may make UMAP
slower to converge on the optimal embedding. "auto" selects between "spectral" and "random" automatically
depending on the size of the dataset.
Values must be one of the following:
* `spectral`
* `pca`
* `tswspectral`
* `random`
* `auto`
Avoid excessive memory use.
For some datasets nearest neighbor computations can consume a lot of memory. If you find
the step is failing due to memory constraints, consider setting this option to `true`.
This approach is more computationally expensive, but avoids excessive memory use. Setting
it to "auto", will enable this mode automatically depending on the size of the dataset.
Values must be one of the following:
* `True`
* `False`
* `auto`
* `None`
Target variable (labels) for supervised dimensionality reduction.
Name of the column that contains your target values (labels).
Weighting factor between features and target.
A value of 0.0 weights entirely on data, and a value of 1.0 weights entirely on target. The
default of 0.5 balances the weighting equally between data and target.
Try to better preserve local densities in the data.
Specifies whether the density-augmented objective of densMAP should be used for optimization.
Turning on this option generates an embedding where the local densities are encouraged to be
correlated with those in the original space.
Strength of local density preservation.
Controls the regularization weight of the density correlation term in densMAP. Higher values
prioritize density preservation over the UMAP objective, and vice versa for values closer to zero.
Setting this parameter to zero is equivalent to running the original UMAP algorithm.
Drop duplicate rows before embedding.
If you have more duplicates than you have `n_neighbors` you can have the identical data points lying
in different regions of your space. It also violates the definition of a metric. This option will
remove duplicates before embedding, and then map the original data points back to the reduced space. Duplicate
data points will be placed in the exact same location as the original data points.
A random number to initialize the algorithm for reproducibility.
Maintain links for n nearest neighbours only in graph.
Values must be in the following range:
```javascript theme={null}
1 ≤ links_top_n ≤ 15
```
Scaling factor for the coordinates.
The maximum (normalized) coordinates in positive and negative X and Y directions. Acts like a zoom, with
a scale of 1 corresponding to zooming out to the maximum (maximal space between nodes), and 0 to the densest
layout.
If set to `"auto"`, will try to determine an appropriate scale taking into account the number of nodes.
If set to `null`, only changes calculated coordinates to ensure they're within the allowed limits (16.000).
null.
string.
number.
Values must be in the following range:
```javascript theme={null}
0 < {_} ≤ 1
```
# layout_igraph
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/layout_igraph
Calculate layout, i.e. node positions, for a network.
Use igraph to create your own layout. Methods and possible parameters are described [here](https://igraph.org/python/doc/igraph.Graph-class.html#layout).
## Usage
The following example shows how the step can be used in a recipe.
The following configuration would allow smaller clusters and consider fewer of the data points as noise:
```stan theme={null}
layout(links) -> (ds.x, ds.y)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
layout_igraph(targets: list[number], *weights: list[number], {
"param": value,
...
}) -> (x: column, y: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column containing link targets. Source is implied in the index.
A numerical column with the x position of the calculated layout.
A numerical column with the y position of the calculated layout.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Algorithm to use.
The name of a supported clustering algorithm (currently allows 'fr', 'fruchterman\_reingold', 'drl', 'sugiyama').
Values must be one of the following:
* `fr`
* `fruchterman_reingold`
* `drl`
* `sugiyama`
Links directed.
Are the links directed (true) or not (false).
Seed.
Seed to initialize the graph.
# layout_network
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/layout_network
Compute a force-directed graph layout with a fast forceAtlas2 implementation.
## Usage
The following example shows how the step can be used in a recipe.
For a weighted layout provide both input columns, e.g.
```stan theme={null}
layout_network(ds.link_targets, ds.link_weights, {scalingRatio: 0.8, linLogMode: false}) -> (ds.x, ds.y)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
layout_network(targets: list[number], *weights: list[number], {
"param": value,
...
}) -> (x: column, y: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column containing link targets. Source is implied in the index.
A column containing the x coordinate of each node's position in the network.
A column containing the y coordinate of each node's position in the network.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Attracts nodes to the center.
Prevents islands from drifting away.
* 0.05
The amount of repulsion.
Greater values lead to a larger and more sparse graph.
* 0.8
Algorithmic "resolution".
Greater values lead to faster execution at the expense of less precise calculations.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ barnesHutTheta ≤ 2.0
```
Prefer authorities over hubs.
Prefer "authorities" (nodes with a high indegree) over hubs (nodes with a high outdegree). Authorities will have more central and hubs more peripheral positions. (default=false).
* False
Usually produces tighter clusters.
Enabling it may also require adjusting the scalingRatio.
* False
Try to avoid overlap between nodes.
* False
The more the better, though it will take longer.
* 300
How much space to (try and) give each node in the final layout.
Values must be in the following range:
```javascript theme={null}
1.0 ≤ nodeSize < inf
```
Links with weights below this value will be ignored.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ ignoreWeightsBelow < inf
```
Normalize weights to the range \[0, 1].
# layout_treemap
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/layout_treemap
Place nodes on the screen using a treemap layout.
Transforms a set of columns (normally categorical) to x and y positions using a treemap layout.
The treemap will be created hierarchically according to the order of columns in the input.
At the lowest level of the treemap, points within the same category may be ordered from top to bottom
and left to right using an optional numerical column.
## Usage
The following example shows how the step can be used in a recipe.
The following configuration transforms the categories into a treemap and uses the numerical column to order the nodes in the final squares.
```stan theme={null}
layout_treemap(ds.salary, ds.position, ds.average_montly_hours, {
"numerical_col": "average_montly_hours",
"node_size": 8
}) -> (ds.x, ds.y)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
layout_treemap(*columns: category|number, {
"param": value,
...
}) -> (x: column, y: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Numerical column with the x position in the graph.
Numerical column with the y position in the graph.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Name of the numerical column.
Used to order the nodes consecutively in the final treemap.
Size of the nodes in pixels.
Values must be in the following range:
```javascript theme={null}
1 ≤ node_size < inf
```
Enable use of version 2 of algorithm.
# link_embeddings
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/link_embeddings
Create network links between rows/nodes calculating the similarity of embeddings (vectors).
Uses Spotify's `Annoy` to perform approximate nearest neighbour search.
## Usage
The following examples show how the step can be used in a recipe.
To link similar embeddings with default configuration
```stan theme={null}
link_embeddings(ds.embedding) -> (ds.targets, ds.weights)
```
To use a similarity cutoff below which similar embeddings won't be connected
```stan theme={null}
link_embeddings(ds.embedding, {"similarity_min": 0.7}) -> (ds.targets, ds.weights)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
link_embeddings(embedding: list[number], {
"param": value,
...
}) -> (targets: column, weights: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A categorical column containing embeddings (numerical vectors/lists). Usually the result of
previously executing a step embed\_\[entity].
A column containing for each row a list of IDs (row numbers) identfying other rows it will be linked to.
A column containing for each row a list of weights identfying the "importance" of each link to
targets identified in the `targets` column.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Number of nearest neighbours to connect to.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_nearest < inf
```
Minimum similarity for connecting two nodes.
Values must be in the following range:
```javascript theme={null}
0 ≤ similarity_min ≤ 1
```
Minimum similarity for connecting two nodes, expressed as a quantile of the similarity distribution.
Values must be in the following range:
```javascript theme={null}
0 ≤ similarity_min_q ≤ 1
```
Number of trees.
Affects the build time and the index size. A larger value will give more accurate results, but will take
longer to create a larger index.
Accuracy multipler.
A larger value will give more accurate results, but will take longer time to return.
Metric to use, only angular supported for now.
Annoy's angular metric is equivalent to sqrt(2\*(1-cos(u,v))), whose max. is sqrt(2\*2) = 2.
I.e. the distance between (1,0) and (-1,0), at maximum angular separation, should be exactly 2
Note that for the weights of the resulting network links Annoy's distances are converted to
similarities in the interval \[0,1].
Values must be one of the following:
* `angular`
* `euclidean`
* `manhattan`
* `hamming`
* `dot`
Used to seed the random number generator, creating deterministic results.
# link_grouped_embeddings
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/link_grouped_embeddings
Create network links calculating the similarity of embeddings (vectors) within groups.
Creates network links only if the row's embeddings belong to the same group.
E.g., text embeddings calculated for different languages are not necessarily compatible (even if they
have the same dimension). Use this step if embeddings in different groups cannot be compared directly.
## Usage
The following example shows how the step can be used in a recipe.
To configure a minimum similarity between embeddings to create a link
```stan theme={null}
link_grouped_embeddings(ds.embedding, ds.group, {
"similarity_min": 0.7
}) -> (ds.targets, ds.weights)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
link_grouped_embeddings(embedding: list[number], grouping: category, {
"param": value,
...
}) -> (targets: column, weights: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A categorical column containing embeddings (numerical vectors/lists). Usually the result of previously executing a step embed\_\[entity].
A categorical column identifying the groups whose embeddings are compatible.
A column containing for each row a list of IDs (row numbers) identfying other rows it will be linked to.
A column containing for each row a list of weights identfying the "importance" of each link to
targets identified in the `targets` column.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Number of nearest embeddings to take into account.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_nearest < inf
```
Minimum similarity for connecting two nodes (similarity ∈ \[0, 1]).
Values must be in the following range:
```javascript theme={null}
0 ≤ similarity_min ≤ 1
```
Minimum similarity for connecting two nodes, expressed as a quantile of the similarity distribution (similarity ∈ \[0, 1]).
Values must be in the following range:
```javascript theme={null}
0 ≤ similarity_min_q ≤ 1
```
Number of trees.
Affects the build time and the index size. A larger value will give more accurate results, but will take
longer to create a larger index.
Accuracy multipler.
A larger value will give more accurate results, but will take longer time to return.
Metric to use, only angular supported for now.
Annoy's angular metric is equivalent to sqrt(2\*(1-cos(u,v))), whose max. is sqrt(2\*2) = 2.
I.e. the distance between (1,0) and (-1,0), at maximum angular separation, should be exactly 2
Note that for the weights of the resulting network links Annoy's distances are converted to similarities in the interval \[0,1].
Values must be one of the following:
* `angular`
* `euclidean`
* `manhattan`
* `hamming`
* `dot`
Used to seed the random number generator, creating deterministic results.
# link_rows
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/link_rows
Create network links using explicit lists of target IDs, weights and other link attributes.
For each row this step iterates over the IDs in the `targets_in` column, and if an ID exists
also in the `source` column, the corresponding rows will be connected, optionally with specified
attributes.
The `targets_in` column may contain one target ID per row, or lists of target IDs. In either case,
any additional attribute columns should be of the same type. I.e. if each row specifies multiple links via
lists in `targets_in`, then attribute columns should also contain lists of the same length, such that each
link can be assigned its corresponding attribute. If the lengths of lists containing target IDs and attributes
do not match, the attributes for links in that row will be missing. If attributes are single-valued (not
containing lists), all links specified in that row will have the same attribute value.
Note that the *types* of values in `source` and `link_targets` identifying the nodes/rows to be linked should also
match. Ideally, either both columns have numeric values or both have string-like (categorical) values. However, as
long as one can be converted safely to the other, linking will work as expected (e.g. source IDs could be specified
as numbers \[0, 1, 2] and target IDs as strings \["3", "2", "1"] without the step failing).
The step will generate at least target and weight columns, as well as another column for each input. If link attribute
columns were passed, the `weight_column` parameter should be used to identify the column containing link weights
(importances). If there is no such column, the parameter value should be `null`, in which case an new weights column
will be generated automatically (see parameters below).
## Usage
The following example shows how the step can be used in a recipe.
In the following example we connect rows/nodes identified in the column `link_source`, to rows/nodes specified in the column `link_targets`, which contains *lists* of such link targets.
Additionally, we use the columns `link_weights` and `links_are_reciprocal` (which contain lists of the same lengths as `targets`), to add attributes to the created links (the weight of the link and whether it is unidirectional or bidirectional).
```stan theme={null}
link_rows(ds.link_source, ds.link_targets, ds.link_weights, ds.links_are_reciprocal) -> (ds.targets_out, ds.weights_out, ds.are_reciprocal_out)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
link_rows(source: number|category, targets_in: number|category|list[number]|list[category], *attrs_in: column, {
"param": value,
...
}) -> (targets_out: column, *attrs_out: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column of (numerical or categorical) IDs identifying the nodes/rows acting as the source of a link.
These need to be compatible with the IDs in the `targets_in` column! E.g. if these are twitter handles,
then the targets must also be twitter handles.
A column containing (potentially lists) of IDs corresponding to link targets.
One ore more optional attributes for the links. Must be lists of the same lengths as `link_targets` if the
latter contains lists. If an attribute column has a single value per row, it is assumed that all targets in
that row have the same attribute value.
A column containing new lists of IDs corresponding to link targets.
If optional inputs were provided, new weight/attribute columns between connected nodes.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Name of the column acting as the weights of the links.
Must refer to one of the optional columns passed to the step. If `null`, an extra
output column will be created containing a weight of 1.0 for each link defined in
the target column (unless a `weight_factor` is applied, in which case the weights
will have the corresponding value, see below).
Multiply link weights by this number.
# link_rows_by_id
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/link_rows_by_id
Create network links using one or more lists of target ids.
For each row this step iterates over the lists of IDs in one or more target columns, and if a ID exists also
in the source column, the corresponding rows will be connected.
Note that while this step allows multiple input columns to be used as link *targets*, it does not allow for the
specification of link *weights*. See the step `link_rows` for creating weighted networks. All link weights will
be set to 1.0 by default. But see the weight\_factor param to specify another constant instead.
## Usage
The following example shows how the step can be used in a recipe.
Given a dataset `ds`, where each row is associated with a twitter user (identified by column `account_id`), the following line connects each of these users with other users specified in columns `reply_ids` and `mention_ids`.
```stan theme={null}
link_rows_by_id(ds.account_id, ds.reply_ids, ds.mention_ids) -> (ds.targets, ds.weights)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
link_rows_by_id(source_id: number|category, *target_ids: number|category|list[number]|list[category], {
"param": value,
...
}) -> (targets: column, weights: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column of IDs corresponding to the nodes/rows acting as the source of a link.
One or more columns of IDs (can be lists) corresponding to the target of a link.
A column containing for each item a list of row numbers identfying all other items it
will be linked to.
A column containing for each item a list of weights identfying the "importance" of each
link to other items identified in the `targets` column (counting how many times a consecutive
pair of items was found together in the sequences).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Multiply link weights by this number.
# link_rows_by_rownum
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/link_rows_by_rownum
Create network links using explicit lists of target row numbers and optional weights.
This step essentially just adds metadata to the input columns to ensure Graphext knows
that these columns define network links and that they belong to the same set of links
(there can be multiple "layers" of links in the same dataset). But it also makes sure
all links are valid. E.g. that they don't refer to rows that don't exist, that attributes
match the number of target rows etc.
## Usage
The following example shows how the step can be used in a recipe.
To simply link rows using a default weight of 1.0
```stan theme={null}
link_rows_by_rownum(ds.targets_in, ds.weights_in) -> (ds.valid_targets, ds.valid_weights)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
link_rows_by_rownum(targets_in: number|list[number], *attrs_in: column, {
"param": value,
...
}) -> (targets_out: column, *attrs_out: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column of lists containing numeric IDs corresponding to the rows acting as the targets of links.
Optional corresponding lists of weights and/or other attributes for those targets.
Column containing new targets.
If optional inputs were provided, new weight/attribute columns between connected nodes.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Name of the column acting as the weights of the links.
Must refer to one of the optional columns passed to the step. If `null`, an extra
output column will be created containing a weight of 1.0 for each link defined in
the target column (unless a `weight_factor` is applied, in which the weights will
have the corresponding value, see below).
Multiply link weights by this number.
If an input column with weights was identified using `weight_column`, the values
in that column will be multiplied by this factor. If no weights were passed in,
the newly added weights will all have this value.
# link_sequence_items
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/link_sequence_items
Create network links between consecutive pairs in a column of sequences.
## Usage
The following example shows how the step can be used in a recipe.
```stan theme={null}
link_sequence_items(ds.items, ds.sequences) -> (ds.targets, ds.weights)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
link_sequence_items(items: number|category, sequences: list[category]|list[number]
) -> (targets: column, weights: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column of items. Should contain unique IDs identifying each item, and corresponding
to the IDs in the sequences column.
Sequences (lists) of items, corresponding to the IDs in the items column.
A column containing for each item a list of row numbers identfying other items it
will be linked to.
A column containing for each item a list of weights identfying the "importance" of each
link to other items identified in the `targets` column (counting how many times a consecutive
pair of items was found together in the sequences).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
This step doesn't expect any configuration.
# link_session_items
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/link_session_items
Link items (e.g. products) in sessions (baskets) if one item makes the presence of the other in the same session more likely.
A link (or association) A->B is created between items A and B if the presence of A makes the presence
of B in the same session N times more likely.
For further details about the algorithm see e.g. [association rule learning](https://en.wikipedia.org/wiki/Association_rule_learning).
## Usage
The following example shows how the step can be used in a recipe.
The following call creates links between pairs of items A and B, if:
* A occurs in at least 7 sessions
* B occurs in at least 25% of sessions containing A
* The presence of A in a session makes the presence of B in the same session at least twice as likely.
Note that the last condition is equivalent to saying that the overall frequency of B in all sessions must be less than 12.5% (half of 25%). In other words, a minimum lift of 2 means that the frequency of B, in sessions already containing A, must be twice the background frequency of B in general.
As an example, the percentage of shopping baskets containing milk (item B) may be 10%. However, amongst those baskets already containing cereals, the percentage containing milk is likely to be higher. If milk occured e.g. in 30% of baskets also having cereals, than the lift of the rule cereal->milk would be 3. The buying of cereal make the buying of milk 3 times more likely.
```stan theme={null}
link_session_items(items.id, sessions.item_ids, {
"min_support": 7,
"min_confidence": 25,
"min_lift": 2
}) -> (items.targets, items.weights)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
link_session_items(items: category|number, sessions: list[category]|list[number], {
"param": value,
...
}) -> (targets: column, weights: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column containing the IDs of items to analyze.
A column containing lists of IDs corresponding to items in the same sessions, basket etc.
A column containing for each item a list of IDs (row numbers) identfying other items it will be linked to.
A column containing for each item a list of weights identfying the "importance" of each link to
other items identified in the `targets` column.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Minimum Support.
Minimum support of a rule antecedent. If it is \< 1 it will be taken as a proportion.
In any other case it will be expected as a positive integer representing the count.
Create link A->B only if A occurred in at least this many sessions.
number.
Values must be in the following range:
```javascript theme={null}
0 < {_} < 1
```
integer.
Values must be in the following range:
```javascript theme={null}
1 ≤ {_} < inf
```
Minimum Confidence.
Expressed as a rule as a percentage.
Include link A->B only if B occurred in at least this percentage of sessions also containing A.
Values must be in the following range:
```javascript theme={null}
0 ≤ min_confidence ≤ 100
```
Minimum Lift.
Expressed as multipler/ratio. Include link A->B only if A makes the presence of B in the same
sessions at least this many times more likely.
Metric for link weight.
Values must be one of the following:
`itemset_support_abs` `itemset_support_pct` `filter_metric_abs` `filter_metric_pct` `antecedent_support_abs` `antecedent_support_pct` `consequent_support_abs` `consequent_support_pct` `rule_confidence_pct` `rule_lift_abs` `rule_lift_pct`
# link_similar_columns
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/link_similar_columns
Calculates all pair-wise column dependencies (by default mutual information).
Create a new network dataset where nodes (rows) represent the original dataset's variables (its columns),
and links represent dependencies between variables (i.e. associations/correlations). By default the step
measures ["mutual information"](https://en.wikipedia.org/wiki/Mutual_information) between variables.
In effect, all pair-wise "correlations" between the original dataset's columns are calculated. A threshold
is then applied to extract only the largest (most interesting) "correlations". These are then translated into
network links between nodes representing the original variables. Each node/row in the new dataset will also
contain information about its correlation with all other nodes (variables).
Note: in this first version, only quantitative and categorical variables will be analyzed (but not tags, lists,
embeddings etc.). You can pass a dataset with arbitrary column types, but those not supported by the selected
correlation method will be ignored in the result.
## Usage
The following examples show how the step can be used in a recipe.
The following example calculates all correlations without filtering, leading to a fully connected correlation network unless some correlations are exactly 0.
```stan theme={null}
links_similar_columns(ds) -> (corrs)
```
The following example removes correlations below the 75th percentile, creating a network only connecting the most correlated variables.
```stan theme={null}
links_similar_columns(ds, {
"min_similarity_quantile": 0.75,
}) -> (corrs)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
link_similar_columns(ds_in: dataset, {
"param": value,
...
}) -> (ds_out: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Input dataset containing arbitrary columns to calculate "correlations" for.
A dataset containing M rows and M+2 columns (where M is the number of columns in the input dataset).
Each row represents a variable in the original dataset, and the columns contain the "correlations" with
the remaining variables. An additional 2 columns ("targets" and "weights") contain links connecting original
variables to other variables they're correlated with.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Correlation method/statistic.
Which statistic to use to measure variable association/correlation.
The default is `"mutual_information"`, which applies scikit-learn\`s k-nearest neighbors implementations (see
[here](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_classif.html) and
[here](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_regression.html)).
It supports both categorical and quantitative variables, is relatively fast, but doesn't have a natural upper bound
(i.e. values are not in the range \[0, 1]).
The `"distance_correlation"` [method](https://en.wikipedia.org/wiki/Distance_correlation) also
supports both categorical and quantitative variables, and has a natural upper bound of 1. It's relatively
slow though, so make sure to select a reasonable value for `n_samples`.
`"distance_correlation_fast"` uses an [optimized implementation](https://dcor.readthedocs.io/en/stable/index.html)
of distance correlation, but only supports quantitative variables.
`"pearson"` is the standard [Pearson correlation coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient),
which also only supports quantitative variables.
Lastly, `"predictive_power"` calculates a version of the [predictive power score (PPS)](https://github.com/8080labs/ppscore/).
This essentially fits a decision tree to predict variable y using only variable x as a predictor, and measures
it performance relative to a dummy/baseline prediction. It supports both categorical and quantitative variables.
Values must be one of the following:
* `mutual_information`
* `distance_correlation`
* `distance_correlation_fast`
* `pearson`
* `predictive_power`
Absolute similarity threshold.
The minimum "correlation" for the creation of a link between two variables.
Similarity threshold expressed as a quantile.
E.g. a value of 0.6 means the bottom 60% of "correlations" will be discarded. Both minima (absolute and
quantile) must be exceeded for a link to be created.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ min_similarity_quantile ≤ 1
```
Replacement for discarded correlations.
The weight of links whose correlations don't pass the minimum threshold. Links with weights of `null` will be
discarded (the default behavior). Can be set e.g. to 0, to generate all possible links.
Number of Samples.
It represents the maximum number of samples to use when measuring "correlations".
Maximum number of links per node.
Ranks the links by weight and keeps only the most similar targets.
The random seed used if applicable for selected `method`.
# link_similar_rows
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/link_similar_rows
Create network links calculating similarity between multidimensional and multitype documents.
Creates a link between each row and the N rows most similar to it. Broadly, the similarity between two rows
is a weighted similarity of the individual columns. The step accepts all data types, i.e. texts, quantitative,
categorical columns etc.
## Usage
The following example shows how the step can be used in a recipe.
```stan theme={null}
link_similar_rows(ds[["bio", "salary", "age", "department"]]) -> (links)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
link_similar_rows(ds: dataset, {
"param": value,
...
}) -> (targets: column, weights: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A dataset containing the columns to be included in the calculation of pair-wise similarities.
Note: a subset of columns can always be selected in a recipe using the ds\[\["column1", "column2", ...]] syntax.
Or to exclude: ds\[!\["column1", "column2", ...]].
A column containing for each row a list of row numbers identfying all other rows it is similar to.
A column containing for each row a list of weights identfying the "importance" of each
link to other rows identified in the `targets` column (identifying how similar the rows are).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Number of similar docs.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_similar_docs < inf
```
Whether to use minhash as a similarity measure.
Number of terms to use.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_terms < inf
```
Minimum term frequency. (For TFIDF).
Values must be in the following range:
```javascript theme={null}
0 ≤ min_term_freq < inf
```
Minimum doc frequency. (For TFIDF).
Values must be in the following range:
```javascript theme={null}
0 ≤ min_doc_freq < inf
```
Maximum doc percentage. (For TFIDF).
Values must be in the following range:
```javascript theme={null}
0 ≤ max_doc_perc ≤ 1
```
Minimum of terms that should match.
Values must be in the following range:
```javascript theme={null}
0 ≤ min_should_match < inf
```
Regex to recognize as string separator.
Languages to use for stopwords.
supports ES, EN and both using commas "ES,EN".
Values must be one of the following:
* `ES`
* `EN`
* `ES,EN`
* `EN,ES`
# merge_links
Source: https://docs.graphext.com/api-docs/analyse/graph_and_map/merge_links
Merge multiple sets of network link columns into a single unified link set.
Takes a dataset containing multiple pairs of link columns (targets + weights) and
combines them into a single pair. This is useful when you have links from different
sources (e.g., embedding similarity links and explicit foreign key links) and want
to visualize or analyze them as a single graph.
Each link pair can optionally be scaled by a weight multiplier, allowing you to
control the relative importance of different link sources.
## Usage
The following examples show how the step can be used in a recipe.
Merge embedding links with FK links, giving FK links 5x weight
```stan theme={null}
merge_links(ds[["semantic_targets", "semantic_weights", "fk_targets", "fk_weights"]], {
"link_pairs": [["semantic_targets", "semantic_weights"], ["fk_targets", "fk_weights"]],
"weight_multipliers": [1.0, 5.0]
}) -> (ds.targets, ds.weights)
```
Merge three link sets with default weights
```stan theme={null}
merge_links(ds[["t1", "w1", "t2", "w2", "t3", "w3"]], {
"link_pairs": [["t1", "w1"], ["t2", "w2"], ["t3", "w3"]]
}) -> (ds.targets, ds.weights)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
merge_links(ds: dataset, {
"param": value,
...
}) -> (targets: column, weights: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A dataset containing the link column pairs to merge.
Merged list of linked row numbers.
Merged list of weights for corresponding targets.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Pairs of \[target\_column, weight\_column] names to merge.
Each entry is a two-element array with the names of the target and weight
columns forming a link pair. E.g., \[\["targets\_a", "weights\_a"], \["targets\_b", "weights\_b"]].
Each item in array.
Each item in array.
Weight multiplier for each link pair.
Optional list of multipliers (one per link pair). Each link pair's weights
will be multiplied by the corresponding value. Defaults to 1.0 for all pairs.
Each item in array.
Values must be in the following range:
```javascript theme={null}
0 ≤ Item < inf
```
# caption_images
Source: https://docs.graphext.com/api-docs/analyse/infer/caption_images
Predict image captions using pretrained DL models.
In its current form the step predicts image captions using [ClipClap](https://github.com/rmokady/CLIP_prefix_caption).
ClipClap first embeds images using the [Clip](https://huggingface.co/docs/transformers/model_doc/clip) model,
which has been pre-trained on 400M image/text pairs to pick out an image's correct caption from a list of candidates. These
images are then projected into the embedding space of the [GPT-2](https://huggingface.co/gpt2) language model, using a
custom model trained for the task. Finally, using this projection as a prefix, the pretained GPT-2 is asked to predict the
next sentence, i.e. the one following the image.
## Usage
The following example shows how the step can be used in a recipe.
The step has no required parameters, so the simplest call is simply
```stan theme={null}
caption_images(ds.image_url) -> (ds.caption)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
caption_images(images: url, {
"param": value,
...
}) -> (caption: text)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column of URLs to images to predict captions for.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Which projection model to use.
The projection model maps embeddings from the pretrained Clip image model, to the pretrained
GPT-2 language model. Select between a multi-layer perceptron ("MLP"), or the faster transformer
("TRF").
Values must be one of the following:
* `TRF`
* `MLP`
Select the parameter set for the model.
The ClipClap authors provide weights for models having been trained either on the
[COCO dataset](https://cocodataset.org/#home) ("coco") or the [ConceptualCaptions](https://ai.google.com/research/ConceptualCaptions/)
dataset ("concept").
Values must be one of the following:
* `coco`
* `concept`
Whether to use beam-search or greedy word prediction.
When enabled, uses a more expensive but "smarter" algorithm to predict the words in the captions.
# classify_text
Source: https://docs.graphext.com/api-docs/analyse/infer/classify_text
Classify texts using any model from the [Hugging Face hub](https://huggingface.co/models).
Note that we do not validate the model name before executing it, so make sure it
corresponds to an existing model in the hub, otherwise the step will fail.
## Usage
The following example shows how the step can be used in a recipe.
To infer the ternary sentiment of tweets using a CardiffNLP model
```stan theme={null}
classify_text(ds.text, {"model": "cardiffnlp/twitter-roberta-base-sentiment"}) -> (ds.text_sentiment)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
classify_text(text: text, {
"param": value,
...
}) -> (class: category)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column of texts to classify.
The inferred class of each text. The labels of individual categories depend on the seleted model,
and/or can be specified manually using the `labels` parameter (see below).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
The name of a model.
This should be the full name (including the organization if applicable) of a model in the
[Hugging Face model hub](https://huggingface.co/models). You can copy it by clicking on the
icon next to the model's name on its dedicated web page.
Note that if the name doesn't correspond to a model existing in the hub the step will fail.
Since there are hundreds if not thousands of potential models, we cannot validate if the
name is correct before executing it.
The specific model version.
Can be a branch name, a tag name, or a commit id. To identify a particular revision, on
a model's webpage (such as [https://huggingface.co/cardiffnlp/twitter-xlm-roberta-base-sentiment-multilingual](https://huggingface.co/cardiffnlp/twitter-xlm-roberta-base-sentiment-multilingual)),
browse to the [Files and versions tab](https://huggingface.co/cardiffnlp/twitter-xlm-roberta-base-sentiment-multilingual/tree/main),
and use the branch or history dropdown menus to see the available branch names or commit IDs.
If not provided, will use the latest (newest) available version (usually from the "main" branch).
Map original model output to human-readable labels.
Unfortunately, many models in Hugging Face are badly configured and output labels like `LABEL_0`,
`LABEL_1`, etc. which isn't very helpful. You can use the "Hosted inference API"
widget on the model's web page to test its output labels. If necessary, use this parameter
to map the default output labels to ones you prefer.
One or more additional parameters.
* E.g. to map ternary sentiment labels
```json theme={null}
"labels": {
"LABEL_0": "negative",
"LABEL_1": "neutral",
"LABEL_2": "positive"
}
```
Minimum probability (score) to accept prediction label.
Class labels with a corresponding probability smaller than this value will be removed
(replaced with NaN, i.e. the missing value).
Values must be in the following range:
```javascript theme={null}
0.0 < min_prob < 1.0
```
How many texts to process simultaneously.
May get ignored when running on CPU.
Values must be in the following range:
```javascript theme={null}
1 ≤ batch_size ≤ 64
```
Number of threads used to feed GPU with texts.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_workers ≤ 4
```
Which CPU/GPU to run model on.
Pass -1 to use CPU, and 0 to use first available GPU. By default, of
when passed `null`, the step will use GPU automatically if one is found
otherwise CPU.
ID of a Hugging Face integration configured in Graphext.
To use a private model from the Hugging Face hub, you need to configure a
Hugging Face "API Key" integration (in the relevant Graphext team > Add Integration
> API KEYS > Add API Key > Hugging Face > paste an access token previously
> configured in your huggingface account). Graphext will automatically assign
> an ID to your integration which gets autocompleted where required (e.g. in the
> recipe editor).
# embed_images
Source: https://docs.graphext.com/api-docs/analyse/infer/embed_images
Embed images using pretrained DL models.
An embedding vector is a numerical representation of an image (or text etc.), such that different numerical components
of the vector capture different dimensions of the image's content. Embeddings can be used, for example, to calculate
the *semantic similarity* between pairs of images (see `link_embeddings`, for example, to create a network of images
connected by similarity).
In its current form the step calculates image embeddings using [Clip](https://huggingface.co/docs/transformers/model_doc/clip),
which has been trained on 400M image/text pairs to pick out an image's correct caption from a list of candidates.
## Usage
The following example shows how the step can be used in a recipe.
The step has no required parameters, so the simplest call is simply
```stan theme={null}
embed_images(ds.image_url) -> (ds.embedding)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
embed_images(images: url, {
"param": value,
...
}) -> (embedding: list[number])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column of URLs to images to calculate embeddings for.
A column of embedding vectors capturing the meaning of each input image.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Whether to normalize embedding vectors (to length/norm of 1.0).
# embed_text_with_model
Source: https://docs.graphext.com/api-docs/analyse/infer/embed_text_with_model
Use language models to calulate an embedding for each text in provided column.
An embedding vector is a numerical representation of a text, such that different numerical components of the vector
capture different dimensions of the text's meaning. Embeddings can be used, for example, to calculate the *semantic similarity*
between pairs of texts. See [`link_embeddings`](https://docs.graphext.com/api-docs/analyse/graph_and_map/create_graph/link_embeddings/),
for example, to create a network of texts connected by similarity.
In this step, embeddings of texts are calculated using pre-trained
[neural language models](https://en.wikipedia.org/wiki/Language»model#Neural_network), especially those using the
popular [transformer architecture](https://huggingface.co/course/chapter1/4) (e.g.
[Bert-based models](https://huggingface.co/transformers/model_doc/bert.html)).
## Things to keep in mind
* Unlike [`embed_text`](https://docs.graphext.com/api-docs/prepare/embed/embed_text/), which uses a different, appropriate spaCy
model for each language in the text column, this step will always use a single model only to calculate embeddings. This
means the model should be multilingual if you have mixed languages, and that otherwise you need to choose the
correct model for your (single) language.
* Each model will be downloaded on the fly before processing the text. This adds a little lag to its execution time (the
bigger the model the longer the download), though for a sufficient number of texts the time spent downloading should not
be significant. Note also, however, that the download, and therefore this step, may fail if the servers of its publisher
are not responsive.
* Since this step potentially supports tens if not hundreds of different models, we cannot provide support or advice on
specific models.
## Usage
The following example shows how the step can be used in a recipe.
To calculate embeddings using a multilingual sentence-bert model (from sentence-transformers):
```stan theme={null}
embed_text_with_model(ds.text, {"collection": "SBERT", "name": "distiluse-base-multilingual-cased-v2"}) -> (ds.embedding)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
embed_text_with_model(text: text, {
"param": value,
...
}) -> (embedding: list[number])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A text column to calculate embeddings for.
A column of embedding vectors capturing the meaning of each input text.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Embed texts using a *Sentence-BERT* model.
Models in this collection (also known as *sentence-transformers*) have been trained specifically for semantic
similarity, i.e. for the purpose of comparing the meaning of texts. Individual models in this collection
can be found here: [https://www.sbert.net/docs/pretrained\_models.html](https://www.sbert.net/docs/pretrained_models.html).
They differ in terms of the language they have been trained on; their size (the bigger the better usually,
but also the slower); as well as their purpose or intended area of application (e.g. it has a specific model
to embed scientific publications).
A specific *Sentence-BERT* model name.
To find a model appropriate for your data or task, check the website of the
[Sentence-BERT model collection](https://www.sbert.net/docs/pretrained_models.html).
* paraphrase-MiniLM-L6-v2
* distiluse-base-multilingual-cased-v2
Whether text embedding vectors should be normalized (to lengths of 1.0).
This may make similarity calculations easier. E.g. we can then use the dot product as a similarity "metric",
instead of the usual cosine angle (which not all downstream functions may support).
How many texts to push through the model at the same time.
Greater values usually mean faster processing (if supported by the model), but also greater use of memory.
Values must be in the following range:
```javascript theme={null}
1 ≤ batch_size < inf
```
Embed texts using a model from the *Hugging Face* hub.
Any pytorch or tensorflow model in [HF's hub](https://huggingface.co/models)
can be used as long as its output contains a [last hidden state](https://huggingface.co/transformers/main_classes/output.html#).
Note however, that using the output embedding of an arbitrary transformer is not always useful, and
specifically may not be approriate for sentence similarity. Rather, these embeddings usually represent the
input for downstream classification tasks instead. A sentence-bert or universal sentence encoder may be more
appopriate in most cases.
A specific *Hugging Face* model name.
To find a model appropriate for your data or task, browse the [Hugging Face model hub](https://huggingface.co/models).
Note that the `name` of a model should include the name of the organization if applicable (e.g.
`"cardiffnlp/"` in the example below).
* cardiffnlp/twitter-xlm-roberta-base
* sentence-transformers/paraphrase-xlm-r-multilingual-v1
Whether text embedding vectors should be normalized (to lengths of 1.0).
This may make similarity calculations easier. E.g. we can then use the dot product as a similarity "metric",
instead of the usual cosine angle (which not all downstream functions may support).
How many texts to push through the model at the same time.
Greater values usually mean faster processing (if supported by the model), but also greater use of memory.
Values must be in the following range:
```javascript theme={null}
1 ≤ batch_size < inf
```
How individual "word" embeddings should be combined.
The output of a transformer contains embeddings for individual words (or sentence pieces, sub-word character
sequences etc.). This parameter determines how these are combined to create a single vector representing the
whole text. This can be the *mean* of individual vectors or the (component-wise) *maximum* (currently pooling
doesn't take the attention mask into account).
Values must be one of the following:
* `mean`
* `max`
# Infer
Source: https://docs.graphext.com/api-docs/analyse/infer/index
| Step | Fast | Description |
| -------------------------------------------------------------------------- | ---- | ----------------------------------------------------------------------------------------- |
| [caption\_images](/api-docs/analyse/infer/caption_images) | | Predict image captions using pretrained DL models |
| [classify\_text](/api-docs/analyse/infer/classify_text) | | Classify texts using any model from the [Hugging Face hub](https://huggingface.co/models) |
| [embed\_images](/api-docs/analyse/infer/embed_images) | | Embed images using pretrained DL models |
| [embed\_text\_with\_model](/api-docs/analyse/infer/embed_text_with_model) | | Use language models to calulate an embedding for each text in provided column |
| [prompt\_ai](/api-docs/analyse/infer/prompt_ai) | | Call OpenAI's models on each row of the dataset for a given prompt |
| [zeroshot\_classify\_text](/api-docs/analyse/infer/zeroshot_classify_text) | | Classify texts using custom labels/categories |
# prompt_ai
Source: https://docs.graphext.com/api-docs/analyse/infer/prompt_ai
Call OpenAI's models on each row of the dataset for a given prompt.
Use any of OpenAI's models on a row-by-row basis. This step doesn't feed the whole dataset into the model, so you won't be able to
perform operations that require more than one row at a time.
It can be used to perform a variety of tasks. Keep in mind that OpenAI's models are generative AI technologies, and thus can give incorrect responses.
It comes with a predefined budget of 5 \$USD, which will prevent the step from executing if it will cost over that budget.
It is advised that you use a filter step first to test the prompt out on a few rows, then launch it on the whole dataset.
Keep in mind our budget is a rough estimate, if you're concerned about cost you should set limits on OpenAI's side.
Your prompt will be configured by using two parameters: 'prompt' and 'response\_format'.
prompt is a text field while response\_format allows you to specify a JSON format for the model's
response, in the format of `{[expected_column]: "description"}`.
Both in the prompt and the response format descriptions you may refer to the row's attributes by using
`${attribute_name}`. Check the examples and parameter documentation below for more information.
???+ info "API integration"
To use this step your team needs to have the *OpenAI* integration configured in Graphext. The corresponding credentials
are required to connect to a third-party API. You can configure API integrations following the `INTEGRATIONS` or `ADD INTEGRATION`
link in the top-left corner of your Team's page, selecting `API keys`, and then the name of the desired third-party service.
First, create an OpenAI account or sign in.
Next, navigate to the API key page and "Create new secret key", optionally naming the key.
Make sure to save this somewhere safe and do not share it with anyone.
Optionally, you can specify the organization the key belongs to.
On [OpenAI](https://platform.openai.com/)'s' page, you can set general budgets for your api key and other settings that may interest you.
## Usage
The following examples show how the step can be used in a recipe.
Specify model
```stan theme={null}
prompt_ai(ds[["Local Address"]], { # contains column 'Local Address'
"integration": "MY_INTEGRATION_ID",
"model": {
"id": "gpt-4.1-mini",
"temperature": 0.2
},
"prompt": "What is the country for ${Local Address}"
}) -> (ds.country)
```
Get attributes from disneyland reviews
```stan theme={null}
prompt_ai(ds[["Review_Text"]],
{
"integration": "open-ai-1-70",
"prompt": "The following is a review from Disneyland. I want you to extract the topics mentioned, the Names of the Disney Characters mentioned and the Names of rides mentioned in this paragraph: '${Review_Text}'. If you do not find or recognize any name of people, company, or rides, simply do not answer anything. NEVER ANSWER WITH 'NULL' VALUE. IMPORTANT: DO NOT ANSWER ANYTHING ELSE IN ANY OTHER CIRCUMSTANCE. DO NOT ANSWER ANYTHING ELSE APART FROM THE JSON",
"model": {
"id": "gpt-4.1-nano"
},
"response_format": {
"topics": "topics mentioned",
"names_of_characters": "names of Disney Characters",
"names_of_rides": "names of rides"
},
"force_format": {
"topics": ["fun", "children", "ride", "food"]
},
"out_types": {
"topics": "list[category]"
}
}) -> (ds.topics,
ds.names_of_characters,
ds.names_of_rides)
```
Classify Tweets
```stan theme={null}
prompt_ai(ds[["authorName", "text"]],
{
"integration": "victoriano-apikey",
"budget": 15,
"model": {
"id": "gpt-4.1-mini",
"temperature": 0.2
},
"prompt": "Classify the following tweet text if it implicitly: criticizes, benefits, is neutral, or is unrelated to each of the main political parties in Spain or any of their members and leaders: ${text} considering the bias of the medium that wrote it with the medium's name: ${authorName}",
"response_format": {
"Clasificacion_PP": "classify the tweet text into only one of these 4 categories related to the Partido Popular (PP), its leader (Álberto Nuñez Feijoo), or any of its members: criticizes PP, benefits PP, neutral for PP, does not mention PP",
"Clasificacion_PSOE": "classify the tweet text into only one of these 4 categories related to the Spanish Socialist Workers' Party (PSOE), its leader (Pedro Sánchez), or any of its members: criticizes PSOE, benefits PSOE, neutral for PSOE, does not mention PSOE",
"Clasificacion_VOX": "classify the tweet text into only one of these 4 categories related to the VOX party, its leader (Santiago Abascal), or any of its members: criticizes VOX, benefits VOX, neutral for VOX, does not mention VOX",
"Clasificacion_SUMAR": "classify the tweet text into only one of these 4 categories related to the SUMAR party, its leader (Yolanda Díez), or any of its members: criticizes SUMAR, benefits SUMAR, neutral for SUMAR, does not mention SUMAR",
"media_bias": "classify the bias of the medium as: right, center, left"
},
"out_types": {
"Clasificacion_PP": "category",
"Clasificacion_PSOE": "category",
"Clasificacion_VOX": "category",
"Clasificacion_SUMAR": "category",
"media_bias": "category"
}
}) -> (ds.Clasificacion_PP,
ds.Clasificacion_PSOE,
ds.Clasificacion_VOX,
ds.Clasificacion_SUMAR,
ds.media_bias)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
prompt_ai(ds: dataset, {
"param": value,
...
}) -> (*outputs: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Dataset to enrich. Make sure it contains the necessary columns.
Number of columns to specify. By default it's set as only one column, of type category.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Main prompt for the API call.
The main body of instructions you wish to perform.
Associated integration.
Prompt instructions for each output column.
Further prompt instructions for each output column.
One or more additional parameters.
Values allowed in each output column.
If provided, values in each column will be restricted.
One or more additional parameters.
Each item in array.
Types for the output column(s).
Desired types for each output column. By default, they will all be categories.
One or more additional parameters.
Values must be one of the following:
`category` `date` `number` `boolean` `url` `sex` `text` `list[number]` `list[category]` `list[url]` `list[boolean]` `list[date]`
Model Configuration.
Configuration for OpenAI's model.
OpenAI model to choose.
Values must be one of the following:
`gpt-4.1` `gpt-4.1-mini` `gpt-4.1-nano` `gpt-5-mini` `gpt-5-nano` `o4-mini`
Temperature. Higher means more creativity, but also makes the model more likely to hallucinate. Lower temperature yields more deterministic results. Ignored for reasoning models (gpt-5-mini, gpt-5-nano, o4-mini).
Values must be in the following range:
```javascript theme={null}
0 ≤ temperature ≤ 1
```
Budget.
If present, the step will not execute if estimated input token cost exceeds this amount in USD.
If max\_out\_tokens is not set, we will minimum of the cost. If it is set, we will give a ceiling.
Actual cost may vary depending on a number of factors like your OpenAI plan. Check your plan before executing.
Maximum output tokens.
If set, each individual response will add to at most this amount. Allows for a budget theorical ceiling to be calculated before executing.
Size of concurrent request at a time.
Lowering this if you have very low rate limits in your plan might prevent empty responses.
Values must be in the following range:
```javascript theme={null}
1 ≤ batch_size ≤ 1000
```
Timeout for requests to OpenAI.
Values must be in the following range:
```javascript theme={null}
1 ≤ timeout < inf
```
# zeroshot_classify_text
Source: https://docs.graphext.com/api-docs/analyse/infer/zeroshot_classify_text
Classify texts using custom labels/categories.
In contrast with [`classify_text`](https://docs.graphext.com/api-docs/prepare/enrich/text/classify_text/),
this step doesn't require a model specifically trained with the given labels. Any model from the
[Hugging Face hub](https://huggingface.co/models) that is compatible with their
[zeroshot classification pipeline](https://huggingface.co/transformers/master/main_classes/pipelines.html#zeroshotclassificationpipeline)
can be used here. By default this is the (English) [`valhalla/distilbart-mnli-12-3`](https://huggingface.co/valhalla/distilbart-mnli-12-3),
for a good trade-off between model size and accuracy. If a multilingual model is needed
you could try e.g. [`joeddav/xlm-roberta-large-xnli`](https://huggingface.co/joeddav/xlm-roberta-large-xnli/).
Note that we do not validate the model name before executing it, so make sure it
corresponds to an existing model in the hub, otherwise the step will fail.
## Usage
The following examples show how the step can be used in a recipe.
E.g., to classify English texts into the three topics `sport`, `politics` and `business`:
```stan theme={null}
zeroshot_classify_text(ds.text, {"labels": ["sport", "politics", "business"]}) -> (ds.topic)
```
Or to try and infer the sentiment of texts in multiple languages:
```stan theme={null}
zeroshot_classify_text(ds.review, {
"labels": ["positive", "negative"],
"template": "The sentiment of this review is {}.",
"model": "joeddav/xlm-roberta-large-xnli"
}) -> (ds.review_sentiment)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
zeroshot_classify_text(text: text, {
"param": value,
...
}) -> (class: category|list[category])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column of texts to classify.
The inferred class of each text. The labels of individual categories are those passed in using the `labels`
parameter below. Depending on whether multilabel classification is activated or not, the output will be
either a simple categorical, or a multilabel categorical column (containing list of categories).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
A list of labels/categories to automatically assign to each text.
This can be somewhat of a black art. As a simple, if perhaps obvious heuristic,
the fewer and less ambiguous the selected categories the faster and most
probably accurate the resulting classification. As the number and ambiguity of
categories increases one can expect less precise results.
Each item in array.
The name of a model.
This should be the full name (including the organization if applicable) of a model in the
[Hugging Face model hub](https://huggingface.co/models). You can copy it by clicking on the
icon next to the model's name on its dedicated web page.
Note that for now Hugging Face only supports models trained on NLI (natural language inference)
tasks in their zeroshot pipeline. These can be recognized usually by mentioning `nli`, `mnli`,
or `xnli` in their name. For further details on zeroshot learning using NLI models see
e.g. [here](https://joeddav.github.io/blog/2020/05/29/ZSL.html#Classification-as-Natural-Language-Inference).
Also, note that if the name doesn't correspond to a model existing in the hub the step will fail.
* joeddav/xlm-roberta-large-xnli
* facebook/bart-large-mnli
A custom hypothesis template.
Hugging Face's NLI-based zeroshot pipeline essentially converts each label into a whole phrase,
and then compares texts againt these phrases to see whether the phrase "agrees" with or "contradicts"
each text. The template parameter can be used to determine *how* a label is converted into a
phrase. The default phrase is `"This text is {}."`, where the curly braces are then replaced
with each label.
If you have texts in a specific language (and if you're using a model appropriate for that single language),
you should probably provide a corresponding template in that language. If you have texts in
mixed languages (and specify a multilingual model), the default template should be fine.
You may also consider using alternative templates specific for your task. E.g. if you're trying to
classify the overall sentiment of product reviews, you may try a template like
`"The sentiment of this review is {}."` (e.g. combined with `"labels": ["positive", "negative"]`).
Whether to allow multiple labels/classes per text.
If this parameter is `false` (default), only the label for the class with the highest probability
will be returned.
If it is `true`, each class will be assigned a probability between 0 and 1. The result will
then contain a list of labels corresponding to all classes with probabilities greater than the
threshold `min_prob` (see below). The classes will be returned in the form of ordered lists,
with the first element being the label of the class with the highest probability.
Only return labels for classes with probability greater than this value.
In single label classification, if even the most probable class falls below this threshold, a missing value
will be returned instead of a label.
When performing multilabel classification, any classes with probabilities below this threshold will simply
be removed from the list of labels in each row. A value of `null` (default), `0.0`, or simply not specifying
this parameter will disable filtering of categories. In this case, the result will contain all classes/labels
for each row, ordered by probability in descending order.
How many texts to process simultaneously.
May get ignored when running on CPU.
Values must be in the following range:
```javascript theme={null}
1 ≤ batch_size ≤ 64
```
# calibrate_classification
Source: https://docs.graphext.com/api-docs/analyse/train_and_predict/calibrate_classification
Calibrate a classification model.
Usually employed after the `train_classification` step to make sure the model's predicted probabilities are well-calibrated.
Note that currently we only support calibration of already fitted models, which should always be performed
on new data not already seen during training. For more information see the
[scikit-learn documentation](https://scikit-learn.org/stable/modules/generated/sklearn.calibration.CalibratedClassifierCV.html).
## Usage
The following example shows how the step can be used in a recipe.
Assuming we have reserved a test set containing data that wasn't used to train the model, we can simply pass it to this step to create a new, calibrated, model:
```stan theme={null}
calibrate(ds_test, "model", {"target": "is_churn", "method": "isotonic"}) -> ("calibrated_model")
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
calibrate_classification(ds: dataset, model: model_classification[ds], {
"param": value,
...
}) -> (model_out: model_classification[ds])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A dataset containing features and target columns for data that has *not* already been used to train the model.
A trained classification model to calibrate.
A zip file containing the calibrated model.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Target variable.
Name of the column that contains your target values (labels).
Calibration method.
Method to use for calibration. `isotonic` is a non-parametric method that fits a piecewise-constant,
strictly increasing function to the predicted probabilities. `sigmoid` (Platt’s method) is a parametric
method that fits a logistic function to the predicted probabilities.
It is not advised to use isotonic calibration with too few calibration samples (much fewer than 1,000) since it tends to overfit.
Values must be one of the following:
* `isotonic`
* `sigmoid`
# explain_predictions
Source: https://docs.graphext.com/api-docs/analyse/train_and_predict/explain_predictions
Explain a prediction model.
Explains the predictions of a trained machine learning model. Currently the only supported method is SHAP, which
provides a unified measure of feature importance and feature effects. For more information see the
[SHAP documentation](https://shap.readthedocs.io/en/latest/).
## Usage
The following example shows how the step can be used in a recipe.
To get json-encoded explanations for a test set:
```stan theme={null}
explain_predictions(ds_test, "my-model", {"positive_class": "True", "verbose": false}) -> (ds.explanation)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
explain_predictions(ds: dataset, model: model_classification[ds], {
"param": value,
...
}) -> (explanation: column, *prediction: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A dataset containing features (but not target column) to calculate explanations for.
A trained model to explain.
A json-encoded, verbose, or list of explanations of the model's predictions for dataset `ds`.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Positive class.
Name/label of the target class to generate explanations for if model is a classifier.
Feature groups.
A dictionary mapping feature names to group names. If provided, explanations will be calculated for each group
of features, rather than for individual features. The resulting explanations will be the sum of the SHAP values
of all features in each group. This can be useful for understanding the overall effect of a group of features.
One or more additional parameters.
Each item in array.
Sign of the SHAP values.
Can be 1 or -1 to focus on SHAP values contributing positively or negatively to the
predictions. This will be taken into account when:
* grouping: only features with the specified sign will be included in the configured groups,
while features not in the group mapping or of a a different sign will be grouped separately
as "uncategorized". Note that this means that the original variables summed in each group
can be different across data points.
* ranking: when selecting the top N features (see below parameter), SHAP values will be ordered
and filtered using the signed values, rather than the absolute values.
0 or null means the sign is ignored when grouping, filtering or ranking SHAP values.
Values must be one of the following:
* `-1`
* `1`
* `0`
* `None`
Top N features.
Number of top features to include in the explanation. If not provided, all features will be included.
Round numerical explanations.
How many decimal places to round the explanations to. If not provided, or `null` will not round.
Output format of the explanations.
If `json`, the default, explanations will be json-encoded. For each row in the dataset, the explanation
consists of an array containing one object for each of the `topn` features, with each object in turn containing
the feature name, the SHAP value, and the feature value (e.g. `"[{'name': 'events': 'data': 5071, 'value': 0.15}, {...}, ...]"`).
The resulting json-encoded output column can be processed further in Graphext using the `extract_json_values` step.
If `verbose`, explanations will be more verbal, using a configurable template to generate a human-readable explanation.
The default format is shown in the `format` parameter below.
If `columns`, the explanations will be returned as separate columns. The first column will contain in each row a list of
the feature names of the `topn` features, sorted descending by SHAP value. The second (optional) column will contain the
corresponding *SHAP* values in the same order. A third (optional) column will contain the corresponding *feature* values.
Values must be one of the following:
* `columns`
* `json`
* `verbose`
Flat or nested records.
If `true`, and the `output` parameter is `"json"`, entries in each output row are flat lists of objects, each containing
the name, the value and SHAP contribution of a feature in the dataset. Additional information, such as the sum of remaining
SHAP contributions (when `topn` or `groups` is set, see `include_tail` below), or the base value, will be included with special
names `""` and `""`, respectively, as if they were features themselved.
If `false`, each output row will contain an object instead, where proper SHAP values are nested under the "shap\_values" key,
while the tail and base value are top-level key-value pairs.
Include tail.
If `true`, the sum of SHAP values of features not included in the `topn` items or groups will also be included in the output.
Include base value.
If `true`, the base value of the model will be included in the output. Note that this value is usually identical
for all rows in the dataset.
Verbal explanation format.
A template string to generate a human-readable explanation (applicable only if parameter `"output": "verbose"`).
The template can contain placeholders for the feature name, the SHAP value, and the feature value (data).
The default format is "(=): ". An even more verbose explanation format could be
`"{name} has a SHAP value of {value} and a feature value of {data}"`, for example. The `topn`
features will be converted using this format and then concatenated using the below `separator`
parameter.
Verbal explanation separator.
A string to separate the explanations of the `topn` features (applicable only if parameter `"verbose": true`).
Explanation space.
The space in which to calculate the explanations. "raw" corresponds to the internal prediction space
of the model, e.g. log-odds in the case of a Catboost classifier. "normalized" will re-normalize the
explanations to the range \[0, 1] for each feature. SHAP values for all features in a single row
will sum to 1.0 in this case. "probability" will convert SHAP values to probabilities by rescaling
the sum of SHAP values for each row such that they sum to the difference between the base probability and
the model's prediction.
Values must be one of the following:
* `raw`
* `normalized`
* `probability`
Base value for to use in explanations.
The base value to use when converting SHAP values to probabilities if the model is a classifier. If not
provided, the mean of the model's predictions on the dataset will be used. Only relevant if `space` is
set to "probability".
Explanation method.
Values must be one of the following:
* `shap`
# Train And Predict
Source: https://docs.graphext.com/api-docs/analyse/train_and_predict/index
| Step | Fast | Description |
| ---------------------------------------------------------------------------------------------------------- | ---- | ------------------------------------------------------------------------------------- |
| [calibrate\_classification](/api-docs/analyse/train_and_predict/calibrate_classification) | | Calibrate a classification model |
| [explain\_predictions](/api-docs/analyse/train_and_predict/explain_predictions) | | Explain a prediction model |
| [predict\_classification](/api-docs/analyse/train_and_predict/predict_classification) | | Use a pretrained classification model to predict new categorical data |
| [predict\_clustering](/api-docs/analyse/train_and_predict/predict_clustering) | | Use a pretrained clustering model to predict new data |
| [predict\_dimensionality\_reduction](/api-docs/analyse/train_and_predict/predict_dimensionality_reduction) | | Use a pretrained model to predict embeddings |
| [predict\_regression](/api-docs/analyse/train_and_predict/predict_regression) | | Use a pretrained model to predict new numerical data |
| [predict\_survival](/api-docs/analyse/train_and_predict/predict_survival) | | Use a pretrained model to predict new data |
| [test\_classification](/api-docs/analyse/train_and_predict/test_classification) | | Evaluate a pretrained classification model on custom test data |
| [test\_classification\_gpu](/api-docs/analyse/train_and_predict/test_classification_gpu) | | Evaluate a pretrained classification model on custom test data |
| [test\_regression](/api-docs/analyse/train_and_predict/test_regression) | | Evaluate a pretrained regression model on custom test data |
| [train\_classification](/api-docs/analyse/train_and_predict/train_classification) | | Train and store a classification model to be loaded at a later point for prediction |
| [train\_classification\_gpu](/api-docs/analyse/train_and_predict/train_classification_gpu) | | Train and store a classification model to be loaded at a later point for prediction |
| [train\_clustering](/api-docs/analyse/train_and_predict/train_clustering) | | Train and store a machine learning model to be loaded at a later point for prediction |
| [train\_dimensionality\_reduction](/api-docs/analyse/train_and_predict/train_dimensionality_reduction) | | Train and store a machine learning model to be loaded at a later point for prediction |
| [train\_regression](/api-docs/analyse/train_and_predict/train_regression) | | Train and store a regression model to be loaded at a later point for prediction |
| [train\_survival](/api-docs/analyse/train_and_predict/train_survival) | | Train and store a survival model to be loaded at a later point for prediction |
# predict_classification
Source: https://docs.graphext.com/api-docs/analyse/train_and_predict/predict_classification
Use a pretrained classification model to predict new categorical data.
Usually employed after the `train_classification` step.
???+ info "Prediction Model"
To use this step successfully you need to make sure the dataset you're predicting on is
as similar as possible to the one the model was trained on. We check that the necessary data
types and columns are present, but you should pay attention to how you handled these in the
recipe the model was generated. Any changes might lead to a significant degradation in
model performance.
## Usage
The following examples show how the step can be used in a recipe.
To only predict the label of the class with the highest probability
```stan theme={null}
predict_classification(ds, "my-model") -> (ds.predicted)
```
To predict the label and its *corresponding* probability
```stan theme={null}
predict_classification(ds, "my-model") -> (ds.predicted, ds.probability)
```
To predict the most likely label but the probability of a *specific* class
```stan theme={null}
predict_classification(ds, "my-model", {"positive_class": "some_label"}) -> (ds.predicted, ds.probability)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
predict_classification(ds: dataset, model: model_classification[ds], {
"param": value,
...
}) -> (*predicted: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Contains the target column and the rest of the columns you wish to use in the model.
File containing the model used to make the prediction.
Column(s) containing the model predictions. If a single output column is provided,
the model will output the predicted class. If two column names are provided, the model
will additionally output the predicted probabilities.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Name of the positive class.
In *binary* classification, usually the class you're most interested in, for example the label/class
corresponding to successful lead conversion in a lead score model, the class corresponding to a
customer who has churned in a churn prediction model, etc.
If provided, will return predicted probabilities for the positive class. If not provided, will return
probabilities for the predicted class (i.e. the 'winning' class with the highest probability).
# predict_clustering
Source: https://docs.graphext.com/api-docs/analyse/train_and_predict/predict_clustering
Use a pretrained clustering model to predict new data.
Usually employed after the `train_clustering` step.
???+ info "Prediction Model"
To use this step successfully you need to make sure the dataset you're predicting on is
as similar as possible to the one the model was trained on. We check that the necessary data
types and columns are present, but you should pay attention to how you handled these in the
recipe the model was generated. Any changes might lead to a significant degradation in
model performance.
## Usage
The following example shows how the step can be used in a recipe.
```stan theme={null}
predict_clustering(ds, model) -> (data.predicted)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
predict_clustering(ds: dataset, model: model_clustering[ds]
) -> (predicted: category)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Contains the target column and the rest of the columns you wish to use in the model.
File containing the model used to make the prediction.
Column containing the model predictions.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
This step doesn't expect any configuration.
# predict_dimensionality_reduction
Source: https://docs.graphext.com/api-docs/analyse/train_and_predict/predict_dimensionality_reduction
Use a pretrained model to predict embeddings.
Usually employed after the `train_dimensionality_reduction` step.
???+ info "Prediction Model"
To use this step successfully you need to make sure the dataset you're predicting on is
as similar as possible to the one the model was trained on. We check that the necessary data
types and columns are present, but you should pay attention to how you handled these in the
recipe the model was generated. Any changes might lead to a significant degradation in
model performance.
## Usage
The following example shows how the step can be used in a recipe.
```stan theme={null}
predict_dimensionality_reduction(ds, model) -> (data.predicted)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
predict_dimensionality_reduction(ds: dataset, model: model_dimensionality_reduction[ds]
) -> (predicted: list[number])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Contains the target column and the rest of the columns you wish to use in the model.
File containing the model used to make the prediction.
Column containing the model predictions.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
This step doesn't expect any configuration.
# predict_regression
Source: https://docs.graphext.com/api-docs/analyse/train_and_predict/predict_regression
Use a pretrained model to predict new numerical data.
Usually employed after the `train_regression` step.
???+ info "Prediction Model"
To use this step successfully you need to make sure the dataset you're predicting on is
as similar as possible to the one the model was trained on. We check that the necessary data
types and columns are present, but you should pay attention to how you handled these in the
recipe the model was generated. Any changes might lead to a significant degradation in
model performance.
## Usage
The following example shows how the step can be used in a recipe.
```stan theme={null}
predict_regression(ds, model) -> (data.predicted)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
predict_regression(ds: dataset, model: model_regression[ds]
) -> (predicted: number)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Contains the target column and the rest of the columns you wish to use in the model.
File containing the model used to make the prediction.
Column containing the model predictions.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
This step doesn't expect any configuration.
# predict_survival
Source: https://docs.graphext.com/api-docs/analyse/train_and_predict/predict_survival
Use a pretrained model to predict new data.
## Usage
The following examples show how the step can be used in a recipe.
Without configuration, the median survival time is returned.
```stan theme={null}
predict_survival(ds, "surv-model") -> (ds.median_survival_time)
```
To predict the survival time at the 0.75 percentile.
```stan theme={null}
predict_survival(ds, "surv-model", {"kind": "percentile", "percentile": 0.75}) -> (ds.survival_time_75th_percentile)
```
To predict the survival function at specific points in time.
```stan theme={null}
predict_survival(ds, "surv-model", {"kind": "survival_function", "times": [1, 2, 3]}) -> (ds.survival_series)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
predict_survival(ds: dataset, model: model_survival[ds], {
"param": value,
...
}) -> (predicted: number)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Contains the target column and the rest of the columns you wish to use in the model.
File containing the model used to make the prediction.
Column containing the model predictions.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Kind of prediction.
`median` returns the median survival time. `percentile` returns the survival time at
the given percentile. `expectation` returns the expected survival time.
`survival_function` returns the whole survival function (one series per sample).
Values must be one of the following:
* `median`
* `percentile`
* `expectation`
* `survival_function`
Percentile when `kind` is set to `percentile`
Values must be in the following range:
```javascript theme={null}
0 ≤ percentile ≤ 1
```
Points in time to predict.
Configures at which points to predict when `kind` is set to `survival_function`.
Either an explicit array of durations, or an object specifying a duration step size and
maximum duration.
array.
Each item in array.
Step size.
Values must be in the following range:
```javascript theme={null}
0 < step < inf
```
Maximum duration.
If not provided, or `null`, the maximum duration in the dataset is used.
Whether to predict remaining time.
Conditions the predictions on known durations. In other words, the prediction is made for
each sample taking into account that the sample has survived up to the duration in this column, and
the prediction is made for the *remaining* time. This applies only to censored samples, where
the event has not been observed. If the event *has* already been observed, on the other hand, predicted
remaining time will be 0 / `null`.
To use this feature, the `target` parameter must also be provided to identify the event and duration
columns in the dataset.
Target variables.
Two names, exactly, corresponding to the target columns that contain in the following order:
1. whether the event was observed (boolean) and
2. the time (duration) to event or censoring (number).
# test_classification
Source: https://docs.graphext.com/api-docs/analyse/train_and_predict/test_classification
Evaluate a pretrained classification model on custom test data.
Usually employed after the `train_classification` step. Useful to potentially
refit the model on a single dataset split and/or predict samples the model
hasn't seen before, calculate errors etc., i.e. for sample-based analysis
of model performance.
## Usage
The following examples show how the step can be used in a recipe.
Assuming we have reserved a test set containing data that wasn't used to train the model, we can simply pass it to this step for evaluation:
```stan theme={null}
test_classification(ds_test, model, {"target": "label"}) -> (ds_test.pred, ds_test.prob, ds_test.error)
```
If the test data is contained in a larger dataset (e.g. along training data), but can be identified using a column indicating the split, we can use the following setup:
```stan theme={null}
test_classification(ds, model, {
"target": "label",
"refit": true,
"split": {
"column": "split_name",
"train_split": "train"
"test_split": "test"
}
}) -> (ds.pred, ds.prob, ds.error)
```
Alternatively, we can create a randomized train/test split on the fly, re-fit the model on the train set, and evaluate on the test set. In this case an additional column will be added to the dataset, indicating the split each row belongs to:
```stan theme={null}
test_classification(ds, model, {
"target": "label",
"refit": true,
"split": {
"test_size": 0.2
}
}) -> (ds.pred, ds.prob, ds.error, ds.split)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
test_classification(ds: dataset, model: model_classification[ds], {
"param": value,
...
}) -> (pred: column, prob: column, error: column, *split: column
)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A dataset containing features and target columns.
Name of trained model to use for prediction.
Column containing the model predictions.
Column containing the model prediction probabilities.
Column containing the model prediction errors.
Optional column identifying the train/test split, if dataset was randomly sampled and model re-fit.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Target variable.
Name of the column that contains your target values (labels).
Name of the positive class.
In *binary* classification, usually the class you're most interested in, for example the label/class
corresponding to successful lead conversion in a lead score model, the class corresponding to a
customer who has churned in a churn prediction model, etc.
If provided, will return predicted probabilities for the positive class. If not provided, will return
probabilities for the predicted class (i.e. the class with the highest probability).
Train/test split configuration.
Identify the splits using an existing column or create a randomized split. In either case,
the model will be refit on the train split and evaluated on the test split.
Size of test split.
The fraction of data used for testing. The remaining data will be used to refit the model.
Values must be in the following range:
```javascript theme={null}
0.0 < test_size < inf
```
Random seed.
Seed used to initialize the random number generator assigning rows to train/test splits.
If none is provided, result will be non-deterministic.
Split column.
Name of the column that contains the split identifiers/names.
Test split identifier.
Value of the split column that identifies the test set. Rows with this value will be used
to evaluate the model. If no `train_split` parameter is provided, the remaining rows will be
used to refit the model before evaluation.
Train split identifier.
Value of the split column that identifies the train set. Rows with this value will be used
to refit the model before evaluation. If not provided, all rows not belonging to the test split
will be used in the refit.
Whether to retrain the model.
If set to `true`, the model will be refit on the train split before evaluation. If set to `false`,
the model will be evaluated on the test split without refitting. If no `split` configuration is provided,
this parameter is ignored.
# test_classification_gpu
Source: https://docs.graphext.com/api-docs/analyse/train_and_predict/test_classification_gpu
Evaluate a pretrained classification model on custom test data.
Usually employed after the `train_classification` step.
???+ info "Prediction Model"
To use this step successfully you need to make sure the dataset you're predicting on is
as similar as possible to the one the model was trained on. We check that the necessary data
types and columns are present, but you should pay attention to how you handled these in the
recipe the model was generated. Any changes might lead to a significant degradation in
model performance.
## Usage
The following examples show how the step can be used in a recipe.
Assuming we have reserved a test set containing data that wasn't used to train the model, we can simply pass it to this step for evaluation:
```stan theme={null}
test_classification(ds_test, model, {"target": "label"}) -> (ds_test.pred, ds_test.prob, ds_test.error)
```
If the test data is contained in a larger dataset (e.g. along training data), but can be identified using a column indicating the split, we can use the following setup:
```stan theme={null}
test_classification(ds, model, {
"target": "label",
"refit": true,
"split": {
"column": "split_name",
"train_split": "train"
"test_split": "test"
}
}) -> (ds.pred, ds.prob, ds.error)
```
Alternatively, we can create a randomized train/test split on the fly, re-fit the model on the train set, and evaluate on the test set. In this case an additional column will be added to the dataset, indicating the split each row belongs to:
```stan theme={null}
test_classification(ds, model, {
"target": "label",
"refit": true,
"split": {
"test_size": 0.2
}
}) -> (ds.pred, ds.prob, ds.error, ds.split)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
test_classification_gpu(ds: dataset, model: model_classification[ds], {
"param": value,
...
}) -> (pred: column, prob: column, error: column, *split: column
)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A dataset containing features and target columns.
Name of trained model to use for prediction.
Column containing the model predictions.
Column containing the model prediction probabilities.
Column containing the model prediction errors.
Optional column identifying the train/test split, if dataset was randomly sampled and model re-fit.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Target variable.
Name of the column that contains your target values (labels).
Name of the positive class.
In *binary* classification, usually the class you're most interested in, for example the label/class
corresponding to successful lead conversion in a lead score model, the class corresponding to a
customer who has churned in a churn prediction model, etc.
If provided, will return predicted probabilities for the positive class. If not provided, will return
probabilities for the predicted class (i.e. the class with the highest probability).
Train/test split configuration.
Identify the splits using an existing column or create a randomized split. In either case,
the model will be refit on the train split and evaluated on the test split.
Size of test split.
The fraction of data used for testing. The remaining data will be used to refit the model.
Values must be in the following range:
```javascript theme={null}
0.0 < test_size < inf
```
Random seed.
Seed used to initialize the random number generator assigning rows to train/test splits.
If none is provided, result will be non-deterministic.
Split column.
Name of the column that contains the split identifiers/names.
Test split identifier.
Value of the split column that identifies the test set. Rows with this value will be used
to evaluate the model. If no `train_split` parameter is provided, the remaining rows will be
used to refit the model before evaluation.
Train split identifier.
Value of the split column that identifies the train set. Rows with this value will be used
to refit the model before evaluation. If not provided, all rows not belonging to the test split
will be used in the refit.
Whether to retrain the model.
If set to `true`, the model will be refit on the train split before evaluation. If set to `false`,
the model will be evaluated on the test split without refitting. If no `split` configuration is provided,
this parameter is ignored.
# test_regression
Source: https://docs.graphext.com/api-docs/analyse/train_and_predict/test_regression
Evaluate a pretrained regression model on custom test data.
Usually employed after the `train_regression` step.
???+ info "Prediction Model"
To use this step successfully you need to make sure the dataset you're predicting on is
as similar as possible to the one the model was trained on. We check that the necessary data
types and columns are present, but you should pay attention to how you handled these in the
recipe the model was generated. Any changes might lead to a significant degradation in
model performance.
## Usage
The following examples show how the step can be used in a recipe.
Assuming we have reserved a test set containing data that wasn't used to train the model, we can simply pass it to this step for evaluation:
```stan theme={null}
test_regression(ds_test, model, {"target": "label"}) -> (ds_test.pred, ds_test.error)
```
If the test data is contained in a larger dataset (e.g. along training data), but can be identified using a column indicating the split, we can use the following setup:
```stan theme={null}
test_regression(ds, model, {
"target": "label",
"refit": true,
"split": {
"column": "split_name",
"train_split": "train"
"test_split": "test"
}
}) -> (ds.pred, ds.error)
```
Alternatively, we can create a randomized train/test split on the fly, re-fit the model on the train set, and evaluate on the test set. In this case an additional column will be added to the dataset, indicating the split each row belongs to:
```stan theme={null}
test_regression(ds, model, {
"target": "label",
"refit": true,
"split": {
"test_size": 0.2
}
}) -> (ds.pred, ds.error, ds.split)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
test_regression(ds: dataset, model: model_regression[ds], {
"param": value,
...
}) -> (pred: column, error: column, *split: column
)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A dataset containing features and target columns.
Name of trained model to use for prediction.
Column containing the model predictions.
Column containing the model prediction errors.
Optional column identifying the train/test split, if dataset was randomly sampled and model re-fit.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Target variable.
Name of the column that contains your target values (labels).
Train/test split configuration.
Identify the splits using an existing column or create a randomized split. In either case,
the model will be refit on the train split and evaluated on the test split.
Size of test split.
The fraction of data used for testing. The remaining data will be used to refit the model.
Values must be in the following range:
```javascript theme={null}
0.0 < test_size < inf
```
Random seed.
Seed used to initialize the random number generator assigning rows to train/test splits.
If none is provided, result will be non-deterministic.
Split column.
Name of the column that contains the split identifiers/names.
Test split identifier.
Value of the split column that identifies the test set. Rows with this value will be used
to evaluate the model. If no `train_split` parameter is provided, the remaining rows will be
used to refit the model before evaluation.
Train split identifier.
Value of the split column that identifies the train set. Rows with this value will be used
to refit the model before evaluation. If not provided, all rows not belonging to the test split
will be used in the refit.
Whether to retrain the model.
If set to `true`, the model will be refit on the train split before evaluation. If set to `false`,
the model will be evaluated on the test split without refitting. If no `split` configuration is provided,
this parameter is ignored.
# train_classification
Source: https://docs.graphext.com/api-docs/analyse/train_and_predict/train_classification
Train and store a classification model to be loaded at a later point for prediction.
The output will consist of a new column with the trained model's predictions on the training data,
as well as a saved and named model file that can be used in other projects for prediction of new data.
Optionally, if a second output column name is provided, the model's predicted probabilities will also be
returned.
A detailed guide on how to configure this step for model tuning and performance evaluation can be found
[here](/guides/model_train_eval/).
## Usage
The following examples show how the step can be used in a recipe.
Train a classification model with default parameters. By default, a Catboost model will be trained, but this can be changed to any of the supported models by specifying the `model` parameter (see below for details):
```stan theme={null}
train_classification(ds, {
"target": "class"
}) -> (ds.predicted, "my-clf")
```
To also return the predicted probabilities, provide a second column name:
```stan theme={null}
train_classification(ds, {
"target": "class"
}) -> (ds.predicted, ds.probs, "my-clf")
```
To be more explicit about which model parameters to use during training, which parameters to optimize (tune) automatically, and how to evaluate the model's performance, the following example shows a complete configuration. It will explicitly select the CatboostClassifier as the model, set `boosting_type` to "ordered", and select the best combination of `learning_rate` and `depth` from the values specified in the `tune: params` configuration. To find the best parameters, it will perform 5-fold cross-validation on each combination, and will use the scorer `f1_weighted` to measure the performance. `accuracy` will also be measured, but only for the purpose of reporting. The best parameter combination will than be evaluated on a single split of the dataset (with 20% of rows used for testing and 80% for training), with metrics selected automatically. Note that the final model will always be re-trained on the whole dataset!
```stan theme={null}
train_classification(ds, {
"target": "label_col",
"model": "CatboostClassifier",
"params": {
"boosting_type": "ordered"
},
"tune": {
"strategy": "grid",
"params": {
"learning_rate": [0.03, 0.1],
"depth": [4, 6, 10]
},
"validate": {
"n_splits": 5,
"metrics": ["f1_weighted", "accuracy"]
},
"scorer": "f1_weighted"
},
"validate": {
"n_splits": 1,
"test_size": 0.2
}
}) -> (ds.predicted, "my-clf")
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
train_classification(ds: dataset, {
"param": value,
...
}) -> (*predicted: column, model: model_classification[ds])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Should contain the target column and the feature columns you wish to use in the model.
One or two columns containing the model's predictions. If two column names are provided, the second column
will contain the model's predicted probabilities.
Zip file containing the trained model and associated information.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Train a [Catboost classifier](https://catboost.ai/docs/).
I.e. gradient boosted decision trees with support for categorical variables and missing values.
Target variable (labels).
Name of the column that contains your target values (labels).
Name of the positive class.
In *binary* classification, usually the class you're most interested in, for example the label/class
corresponding to successful lead conversion in a lead score model, the class corresponding to a
customer who has churned in a churn prediction model, etc.
If provided, will automaticall measure the performance (accuracy, precision, recall) of the model on this
class, in addition to averages across all classes. If not provided, only summary metrics will be reported.
Maximum number of classes in the target variable.
If there are more classes than this, the least frequent classes will be grouped together into a single class
called "others". Reducing the number of classes in the target variable can help improve model performance,
especially when the number of classes is very large, some classes are very rare, or the dataset doesn't have
sufficient samples for all classes. Raising this significantly might lead to much longer training times.
Values must be in the following range:
```javascript theme={null}
2 ≤ max_classes ≤ 100
```
Importance of each feature in the model.
Whether and how to measure each feature's contribution to the model's predictions. The higher the value,
the more important the feature was in the model. Only relative values are meaningful, i.e. the importance
of a feature relative to other features in the model.
Also note that feature importance is usually meaningful only for models that fit the data well.
The default (`null`, `true` or `"native"`) uses the classifier's native feature importance measure, e.g.
[prediction-value-change](https://catboost.ai/en/docs/concepts/fstr#regular-feature-importance) in the case
of Catboost, [Gini importance](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier.feature_importances_)
in the case of scikit-learn's DecisionTreeClassifier, and the mean of absolute coefficients in the case of
logistic regression.
When set to `"permutation"`, uses [permutation importance](https://scikit-learn.org/stable/modules/permutation_importance.html),
i.e. measures the decrease in model score when a single feature's values are randomly shuffled. This is
considerably slower than native feature importance (the model needs to be evaluated an additional k\*n times,
where k is the number of features and n the number of repetitions to average over). On the positive side it is
model-agnostic and doesn't suffer from bias towards high cardinality features (like some tree-based feature
importances). On the negative side, it can be sensitive to strongly correlated features, as the unshuffled
correlated variable is still available to the model when shuffling the original variable.
When set to `false`, no feature importance will be calculated.
Values must be one of the following:
* `True`
* `False`
* `native`
* `permutation`
* `null`
Toggle encoding of feature columns.
When enabled, Graphext will auto-convert any column types to the numeric type before
fitting the model. How this conversion is done can be configured using the `feature_encoder`
option below.
If disabled, any model trained in this step will assume that input data
is already in an appropriate format (e.g. numerical and not containing any missing values).
Configures encoding of feature columns.
By default (`null`), Graphext chooses automatically how to convert any column types the model
may not understand natively to a numeric type.
A configuration object can be passed instead to overwrite specific parameter values with respect
to their default values.
Numeric encoder.
Configures encoding of numeric features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Further parameters passed to the `scaler` function.
Details depend no the particular scaler used.
Boolean encoder.
Configures encoding of boolean features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Ordinal encoder.
Configures encoding of categorical features that have a natural order.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Category encoder.
May contain either a single configuration for all categorical variables, or two different configurations
for low- and high-cardinality variables. For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for categories with fewer than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Multilabel encoder.
Configures encoding of multivalued categorical features (variable length lists of categories,
or the semantic type `list[category]` for short). May contain either a single configuration for
all multilabel variables, or two different configurations for low- and high-cardinality variables.
For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for mulitabel columns with fewer than `cardinality_threshold` unique categories/labels.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Datetime encoder.
Configures encoding of datetime (timestamp) features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
A list of numerical components to extract.
Will create one numeric column for each component.
Each item in array.
Values must be one of the following:
`day` `dayofweek` `dayofyear` `hour` `minute` `month` `quarter` `season` `second` `week` `weekday` `weekofyear` `year`
A list of cyclical time features to extract.
"Cycles" are numerical transformations of features that should be represented on a circle. E.g. months,
ranging from 1 to 12, should be arranged such that 12 and 1 are next to each other, rather than on
opposite ends of a linear scale. We represent such cyclical time features on a circle by creating two
columns for each original feature: the sin and cos of the numerical feature after appropriate scaling.
Each item in array.
Values must be one of the following:
* `day`
* `dayofweek`
* `dayofyear`
* `hour`
* `month`
Whether to include the epoch as new feature (seconds since 01/01/1970).
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Embedding/vector encoder.
Configures encoding of multivalued numerical features (variable length lists of numbers, i.e. vectors, or the semantic type `list[number]` for short).
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Text encoder.
Configures encoding of text (natural language) features. Currently only allows
[Tf-Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) embeddings to represent texts. If you wish
to use other embeddings, e.g. semantic, Word2Vec etc., transform your text column first using
another step, and then use that result instead of the original texts.
Texts are *excluded* by default from the overall encoding of the dataset. See parameter
`include_text_features` below to active it.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Parameters to be passed to the text encoder (Tf-Idf parameters only for now).
See [scikit-learn's documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)
for detailed parameters and their explanation.
How many output features to generate.
The resulting Tf-Idf vectors will be reduced to these many dimensions (columns) using scikit-learn's
[truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
This performs a kind of [latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
By default we will reduce to 200 components.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_components ≤ 1024
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Whether to include or ignore text columns during the processing of input data.
Enabling this will convert texts to their TfIdf representation. Each text will be
converted to an N-dimensional vector in which each component measures the relative
"over-representation" of a specific word (or n-gram) relative to its overall
frequency in the whole dataset. This is disabled by default because it will
often be better to convert texts explicitly using a previous step, such as
`embed_text` or `embed_text_with_model`.
CatBoost configuration parameters.
You can check the official documentation for more details about Catboost's parameters [here](https://catboost.ai/en/docs/references/training-parameters/).
The maximum depth of the tree.
Values must be in the following range:
```javascript theme={null}
2 ≤ depth ≤ 16
```
Number of iterations.
The maximum number of trees that can be built when solving machine learning problems. When using other parameters that limit the number of iterations, the final number of trees may be less than the number specified in this parameter.
Values must be in the following range:
```javascript theme={null}
1 ≤ iterations < inf
```
Maximum cardinality of variables to be one-hot encoded.
Use one-hot encoding for all categorical features with a number of different values less than or equal to this value. Other variables will be target-encoded. Note that one-hot encoding is faster than the alternatives, so decreasing this value makes it more likely slower methods will be used. See [CatBoost details](https://catboost.ai/docs/concepts/algorithm-main-stages_cat-to-numberic.html) for further information.
Values must be in the following range:
```javascript theme={null}
2 ≤ one_hot_max_size < inf
```
The maximum number of features that can be combined when transforming categorical variables.
Each resulting combination consists of one or more categorical features and can optionally contain binary features in the following form: “numeric feature > value”.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_ctr_complexity ≤ 4
```
Coefficient at the L2 regularization term of the cost function.
Values must be in the following range:
```javascript theme={null}
0.0 < l2_leaf_reg < inf
```
The number of splits for numerical features.
Values must be in the following range:
```javascript theme={null}
1 ≤ border_count ≤ 65535
```
The amount of randomness to use for scoring splits.
Use this parameter to avoid overfitting the model. The value multiplies the variance of a random variable (with
zero mean) that is added to the score used to select splits when a tree is grown.
Values must be in the following range:
```javascript theme={null}
0 < random_strength < inf
```
The method for processing missing values in the input dataset.
Possible values:
* “Forbidden”:
Missing values are not supported, their presence is interpreted as an error.
* “Min”:
Missing values are processed as the minimum value (less than all other values) for the feature.
It is guaranteed that a split that separates missing values from all other values is considered
when selecting trees.
* “Max”:
Missing values are processed as the maximum value (greater than all other values) for the feature.
It is guaranteed that a split that separates missing values from all other values is considered when
selecting trees.
Using the Min or Max value of this parameter guarantees that a split between missing values and other
values is considered when selecting a new split in the tree.
Values must be one of the following:
* `Forbidden`
* `Min`
* `Max`
Boosting type.
Boosting scheme. Possible values are
* Ordered: Usually provides better quality on small datasets, but it may be slower than the Plain scheme.
* Plain: The classic gradient boosting scheme.
Values must be one of the following:
* `Ordered`
* `Plain`
Random subspace method.
The percentage of features to use at each split selection, when features are selected over again at random. The value `null` is equivalent to 1.0 (all features). You can set this to values \< 1.0 when the dataset has many features (e.g. > 20) to speed up training.
Values must be in the following range:
```javascript theme={null}
0 < rsm ≤ 1.0
```
The random seed used for training.
Whether and how to limit memory usage.
Select the maximum Ram used using strings like "2GB" or "100mb" (non case\_sensitive).
Whether and how to assign weights to different predicted classes.
The options are:
* null: No class weighting
* Balanced: Inversely proportional to the number of samples/rows in each class
* SqrtBalanced: Using the square root of the "Balanced" option.
Values must be one of the following:
* `Balanced`
* `SqrtBalanced`
* `None`
* `None`
Configure model validation.
Allows evaluation of model performance via [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html)
using custom metrics. If not specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_splits < inf
```
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [k-fold cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#k-fold)
to split the dataset. E.g. if `n_splits` is 5, the dataset will be split into 5 equal-sized parts.
For five iterations four parts will then be used for training and the remaining part for testing.
If `test_size` is a number between 0 and 1, in contrast, validation is done using a
[shuffle-split](https://scikit-learn.org/stable/modules/cross_validation.html#shufflesplit)
approach. Here, instead of splitting the data into `n_splits` equal parts up front, in each iteration
we randomize the data and sample a proportion equal to `test_size` to use for evaluation and the remaining
rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
Whether to split the data by time.
Most recent data will be used for testing and previous data for training. Assumes data is passed
already sorted ascending by time.
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `roc_auc` `roc_auc_ovr` `roc_auc_ovo` `roc_auc_ovr_weighted` `roc_auc_ovo_weighted`
Configure hypertuning.
Configures the optimization of model hyper-parameters via cross-validated grid- or randomized search.
Which search strategy to use for optimization.
Grid search explores all possible combinations of parameters specified in `params`.
Randomized search, on the other hand, randomly samples `iterations` parameter combinations
from the distributions specified in `params`.
Values must be one of the following:
* `grid`
* `random`
How many randomly sampled parameter combinations to test in randomized search.
Values must be in the following range:
```javascript theme={null}
1 < iterations < inf
```
Configure model validation.
Allows evaluation of model performance via [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html)
using custom metrics. If not specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_splits < inf
```
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [k-fold cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#k-fold)
to split the dataset. E.g. if `n_splits` is 5, the dataset will be split into 5 equal-sized parts.
For five iterations four parts will then be used for training and the remaining part for testing.
If `test_size` is a number between 0 and 1, in contrast, validation is done using a
[shuffle-split](https://scikit-learn.org/stable/modules/cross_validation.html#shufflesplit)
approach. Here, instead of splitting the data into `n_splits` equal parts up front, in each iteration
we randomize the data and sample a proportion equal to `test_size` to use for evaluation and the remaining
rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
Whether to split the data by time.
Most recent data will be used for testing and previous data for training. Assumes data is passed
already sorted ascending by time.
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `roc_auc` `roc_auc_ovr` `roc_auc_ovo` `roc_auc_ovr_weighted` `roc_auc_ovo_weighted`
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `roc_auc` `roc_auc_ovr` `roc_auc_ovo` `roc_auc_ovr_weighted` `roc_auc_ovo_weighted`
The parameter values to explore.
Allows tuning of any and all of the parameters that can be set also as constants in the
"params" attribute.
Keys in this object should be strings identifying parameter names, and values should be
*lists* of values to explore for that parameter. E.g. `"depth": [3, 5, 7]`.
List of depths values to explore.
The maximum depth of the tree.
Values must be in the following range:
```javascript theme={null}
2 ≤ Item ≤ 16
```
List of iterations values to explore.
Number of iterations.
The maximum number of trees that can be built when solving machine learning problems. When using other parameters that limit the number of iterations, the final number of trees may be less than the number specified in this parameter.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item < inf
```
List of values configuring max cardinality for one-hot encoding.
Maximum cardinality of variables to be one-hot encoded.
Use one-hot encoding for all categorical features with a number of different values less than or equal to this value. Other variables will be target-encoded. Note that one-hot encoding is faster than the alternatives, so decreasing this value makes it more likely slower methods will be used. See [CatBoost details](https://catboost.ai/docs/concepts/algorithm-main-stages_cat-to-numberic.html) for further information.
Values must be in the following range:
```javascript theme={null}
2 ≤ Item < inf
```
List of values configuring variable combination complexity.
The maximum number of features that can be combined when transforming categorical variables.
Each resulting combination consists of one or more categorical features and can optionally contain binary features in the following form: “numeric feature > value”.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item ≤ 4
```
List of leaf regularization strengths.
Coefficient at the L2 regularization term of the cost function.
Values must be in the following range:
```javascript theme={null}
0.0 < Item < inf
```
List of border counts.
The number of splits for numerical features.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item ≤ 65535
```
List of random strengths.
The amount of randomness to use for scoring splits.
Use this parameter to avoid overfitting the model. The value multiplies the variance of a random variable (with
zero mean) that is added to the score used to select splits when a tree is grown.
Values must be in the following range:
```javascript theme={null}
0 < Item < inf
```
Seed for random number generator ensuring reproducibility.
Values must be in the following range:
```javascript theme={null}
0 ≤ seed < inf
```
Sort the data before training.
If the data is not already sorted by time, you can sort it here. This is useful when you want to split the data
by time, for example to train on older data and test on newer data (see the `time_split` parameter in validation
configurations). If the data is already sorted by time, you can ignore this parameter.
One or more column to sort by.
Each item in array.
Sort order.
Whether to sort in ascending or descending order. If the single value `true` is provided, or no value is specified,
all columns will be sorted in ascending order. If a single `false` is provided, all columns will be sorted in descending
order. If an array of booleans is provided, each column will be sorted according to the corresponding boolean value.
Each item in array.
Train a [Histogram-based gradient-boosting classification tree](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.HistGradientBoostingClassifier.html).
Scikit-learn native boosted trees similar to Catboost or LightGBM, with support for categorical variables,
missing values and early stopping.
Target variable (labels).
Name of the column that contains your target values (labels).
Name of the positive class.
In *binary* classification, usually the class you're most interested in, for example the label/class
corresponding to successful lead conversion in a lead score model, the class corresponding to a
customer who has churned in a churn prediction model, etc.
If provided, will automaticall measure the performance (accuracy, precision, recall) of the model on this
class, in addition to averages across all classes. If not provided, only summary metrics will be reported.
Maximum number of classes in the target variable.
If there are more classes than this, the least frequent classes will be grouped together into a single class
called "others". Reducing the number of classes in the target variable can help improve model performance,
especially when the number of classes is very large, some classes are very rare, or the dataset doesn't have
sufficient samples for all classes. Raising this significantly might lead to much longer training times.
Values must be in the following range:
```javascript theme={null}
2 ≤ max_classes ≤ 100
```
Importance of each feature in the model.
Whether and how to measure each feature's contribution to the model's predictions. The higher the value,
the more important the feature was in the model. Only relative values are meaningful, i.e. the importance
of a feature relative to other features in the model.
Also note that feature importance is usually meaningful only for models that fit the data well.
The default (`null`, `true` or `"native"`) uses the classifier's native feature importance measure, e.g.
[prediction-value-change](https://catboost.ai/en/docs/concepts/fstr#regular-feature-importance) in the case
of Catboost, [Gini importance](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier.feature_importances_)
in the case of scikit-learn's DecisionTreeClassifier, and the mean of absolute coefficients in the case of
logistic regression.
When set to `"permutation"`, uses [permutation importance](https://scikit-learn.org/stable/modules/permutation_importance.html),
i.e. measures the decrease in model score when a single feature's values are randomly shuffled. This is
considerably slower than native feature importance (the model needs to be evaluated an additional k\*n times,
where k is the number of features and n the number of repetitions to average over). On the positive side it is
model-agnostic and doesn't suffer from bias towards high cardinality features (like some tree-based feature
importances). On the negative side, it can be sensitive to strongly correlated features, as the unshuffled
correlated variable is still available to the model when shuffling the original variable.
When set to `false`, no feature importance will be calculated.
Values must be one of the following:
* `True`
* `False`
* `native`
* `permutation`
* `null`
Toggle encoding of feature columns.
When enabled, Graphext will auto-convert any column types to the numeric type before
fitting the model. How this conversion is done can be configured using the `feature_encoder`
option below.
If disabled, any model trained in this step will assume that input data
is already in an appropriate format (e.g. numerical and not containing any missing values).
Configures encoding of feature columns.
By default (`null`), Graphext chooses automatically how to convert any column types the model
may not understand natively to a numeric type.
A configuration object can be passed instead to overwrite specific parameter values with respect
to their default values.
Numeric encoder.
Configures encoding of numeric features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Further parameters passed to the `scaler` function.
Details depend no the particular scaler used.
Boolean encoder.
Configures encoding of boolean features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Ordinal encoder.
Configures encoding of categorical features that have a natural order.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Category encoder.
May contain either a single configuration for all categorical variables, or two different configurations
for low- and high-cardinality variables. For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for categories with fewer than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Multilabel encoder.
Configures encoding of multivalued categorical features (variable length lists of categories,
or the semantic type `list[category]` for short). May contain either a single configuration for
all multilabel variables, or two different configurations for low- and high-cardinality variables.
For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for mulitabel columns with fewer than `cardinality_threshold` unique categories/labels.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Datetime encoder.
Configures encoding of datetime (timestamp) features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
A list of numerical components to extract.
Will create one numeric column for each component.
Each item in array.
Values must be one of the following:
`day` `dayofweek` `dayofyear` `hour` `minute` `month` `quarter` `season` `second` `week` `weekday` `weekofyear` `year`
A list of cyclical time features to extract.
"Cycles" are numerical transformations of features that should be represented on a circle. E.g. months,
ranging from 1 to 12, should be arranged such that 12 and 1 are next to each other, rather than on
opposite ends of a linear scale. We represent such cyclical time features on a circle by creating two
columns for each original feature: the sin and cos of the numerical feature after appropriate scaling.
Each item in array.
Values must be one of the following:
* `day`
* `dayofweek`
* `dayofyear`
* `hour`
* `month`
Whether to include the epoch as new feature (seconds since 01/01/1970).
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Embedding/vector encoder.
Configures encoding of multivalued numerical features (variable length lists of numbers, i.e. vectors, or the semantic type `list[number]` for short).
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Text encoder.
Configures encoding of text (natural language) features. Currently only allows
[Tf-Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) embeddings to represent texts. If you wish
to use other embeddings, e.g. semantic, Word2Vec etc., transform your text column first using
another step, and then use that result instead of the original texts.
Texts are *excluded* by default from the overall encoding of the dataset. See parameter
`include_text_features` below to active it.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Parameters to be passed to the text encoder (Tf-Idf parameters only for now).
See [scikit-learn's documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)
for detailed parameters and their explanation.
How many output features to generate.
The resulting Tf-Idf vectors will be reduced to these many dimensions (columns) using scikit-learn's
[truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
This performs a kind of [latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
By default we will reduce to 200 components.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_components ≤ 1024
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Whether to include or ignore text columns during the processing of input data.
Enabling this will convert texts to their TfIdf representation. Each text will be
converted to an N-dimensional vector in which each component measures the relative
"over-representation" of a specific word (or n-gram) relative to its overall
frequency in the whole dataset. This is disabled by default because it will
often be better to convert texts explicitly using a previous step, such as
`embed_text` or `embed_text_with_model`.
Histogram-based gradient boosting configuration parameters.
You can check the official documentation for more details about the model's parameters [here](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.HistGradientBoostingClassifier.html).
Whether to assign weights to different predicted classes.
If `null` (default), applies no class weighting. If "balanced", assigns weights inversely proportional to the number of samples/rows in each class.
Values must be one of the following:
* `balanced`
* `None`
Configure model validation.
Allows evaluation of model performance via [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html)
using custom metrics. If not specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_splits < inf
```
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [k-fold cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#k-fold)
to split the dataset. E.g. if `n_splits` is 5, the dataset will be split into 5 equal-sized parts.
For five iterations four parts will then be used for training and the remaining part for testing.
If `test_size` is a number between 0 and 1, in contrast, validation is done using a
[shuffle-split](https://scikit-learn.org/stable/modules/cross_validation.html#shufflesplit)
approach. Here, instead of splitting the data into `n_splits` equal parts up front, in each iteration
we randomize the data and sample a proportion equal to `test_size` to use for evaluation and the remaining
rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
Whether to split the data by time.
Most recent data will be used for testing and previous data for training. Assumes data is passed
already sorted ascending by time.
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `roc_auc` `roc_auc_ovr` `roc_auc_ovo` `roc_auc_ovr_weighted` `roc_auc_ovo_weighted`
Configure hypertuning.
Configures the optimization of model hyper-parameters via cross-validated grid- or randomized search.
Which search strategy to use for optimization.
Grid search explores all possible combinations of parameters specified in `params`.
Randomized search, on the other hand, randomly samples `iterations` parameter combinations
from the distributions specified in `params`.
Values must be one of the following:
* `grid`
* `random`
How many randomly sampled parameter combinations to test in randomized search.
Values must be in the following range:
```javascript theme={null}
1 < iterations < inf
```
Configure model validation.
Allows evaluation of model performance via [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html)
using custom metrics. If not specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_splits < inf
```
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [k-fold cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#k-fold)
to split the dataset. E.g. if `n_splits` is 5, the dataset will be split into 5 equal-sized parts.
For five iterations four parts will then be used for training and the remaining part for testing.
If `test_size` is a number between 0 and 1, in contrast, validation is done using a
[shuffle-split](https://scikit-learn.org/stable/modules/cross_validation.html#shufflesplit)
approach. Here, instead of splitting the data into `n_splits` equal parts up front, in each iteration
we randomize the data and sample a proportion equal to `test_size` to use for evaluation and the remaining
rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
Whether to split the data by time.
Most recent data will be used for testing and previous data for training. Assumes data is passed
already sorted ascending by time.
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `roc_auc` `roc_auc_ovr` `roc_auc_ovo` `roc_auc_ovr_weighted` `roc_auc_ovo_weighted`
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `roc_auc` `roc_auc_ovr` `roc_auc_ovo` `roc_auc_ovr_weighted` `roc_auc_ovo_weighted`
The parameter values to explore.
Allows tuning of parameters that can be set also as constants in the "params" attribute.
Keys in this object should be strings identifying parameter names, and values should be
*lists* of values to explore for that parameter. E.g. `"depth": [3, 5, 7]`.
List of depths values to explore.
The maximum depth of the tree.
Values must be in the following range:
```javascript theme={null}
2 ≤ Item ≤ 16
```
List of iterations values to explore.
Number of iterations.
The maximum number of trees that can be built when solving machine learning problems. When using other parameters that limit the number of iterations, the final number of trees may be less than the number specified in this parameter.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item < inf
```
List of values configuring max cardinality for one-hot encoding.
Maximum cardinality of variables to be one-hot encoded.
Use one-hot encoding for all categorical features with a number of different values less than or equal to this value. Other variables will be target-encoded. Note that one-hot encoding is faster than the alternatives, so decreasing this value makes it more likely slower methods will be used. See [CatBoost details](https://catboost.ai/docs/concepts/algorithm-main-stages_cat-to-numberic.html) for further information.
Values must be in the following range:
```javascript theme={null}
2 ≤ Item < inf
```
List of values configuring variable combination complexity.
The maximum number of features that can be combined when transforming categorical variables.
Each resulting combination consists of one or more categorical features and can optionally contain binary features in the following form: “numeric feature > value”.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item ≤ 4
```
List of leaf regularization strengths.
Coefficient at the L2 regularization term of the cost function.
Values must be in the following range:
```javascript theme={null}
0.0 < Item < inf
```
List of border counts.
The number of splits for numerical features.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item ≤ 65535
```
List of random strengths.
The amount of randomness to use for scoring splits.
Use this parameter to avoid overfitting the model. The value multiplies the variance of a random variable (with
zero mean) that is added to the score used to select splits when a tree is grown.
Values must be in the following range:
```javascript theme={null}
0 < Item < inf
```
Seed for random number generator ensuring reproducibility.
Values must be in the following range:
```javascript theme={null}
0 ≤ seed < inf
```
Sort the data before training.
If the data is not already sorted by time, you can sort it here. This is useful when you want to split the data
by time, for example to train on older data and test on newer data (see the `time_split` parameter in validation
configurations). If the data is already sorted by time, you can ignore this parameter.
One or more column to sort by.
Each item in array.
Sort order.
Whether to sort in ascending or descending order. If the single value `true` is provided, or no value is specified,
all columns will be sorted in ascending order. If a single `false` is provided, all columns will be sorted in descending
order. If an array of booleans is provided, each column will be sorted according to the corresponding boolean value.
Each item in array.
Trains a logistic regression.
The specific kind of logistic regression trained here uses "elastic net" regularization, which
allows for a blend of ridge and lasso penalties to prevent overfitting. The mix as well as the
strength of this regularization is automatically tuned using 5-fold cross-validation. See
[sklearn's LogisticRegressionCV](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegressionCV.html)
for further details.
Target variable (labels).
Name of the column that contains your target values (labels).
Name of the positive class.
In *binary* classification, usually the class you're most interested in, for example the label/class
corresponding to successful lead conversion in a lead score model, the class corresponding to a
customer who has churned in a churn prediction model, etc.
If provided, will automaticall measure the performance (accuracy, precision, recall) of the model on this
class, in addition to averages across all classes. If not provided, only summary metrics will be reported.
Maximum number of classes in the target variable.
If there are more classes than this, the least frequent classes will be grouped together into a single class
called "others". Reducing the number of classes in the target variable can help improve model performance,
especially when the number of classes is very large, some classes are very rare, or the dataset doesn't have
sufficient samples for all classes. Raising this significantly might lead to much longer training times.
Values must be in the following range:
```javascript theme={null}
2 ≤ max_classes ≤ 100
```
Importance of each feature in the model.
Whether and how to measure each feature's contribution to the model's predictions. The higher the value,
the more important the feature was in the model. Only relative values are meaningful, i.e. the importance
of a feature relative to other features in the model.
Also note that feature importance is usually meaningful only for models that fit the data well.
The default (`null`, `true` or `"native"`) uses the classifier's native feature importance measure, e.g.
[prediction-value-change](https://catboost.ai/en/docs/concepts/fstr#regular-feature-importance) in the case
of Catboost, [Gini importance](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier.feature_importances_)
in the case of scikit-learn's DecisionTreeClassifier, and the mean of absolute coefficients in the case of
logistic regression.
When set to `"permutation"`, uses [permutation importance](https://scikit-learn.org/stable/modules/permutation_importance.html),
i.e. measures the decrease in model score when a single feature's values are randomly shuffled. This is
considerably slower than native feature importance (the model needs to be evaluated an additional k\*n times,
where k is the number of features and n the number of repetitions to average over). On the positive side it is
model-agnostic and doesn't suffer from bias towards high cardinality features (like some tree-based feature
importances). On the negative side, it can be sensitive to strongly correlated features, as the unshuffled
correlated variable is still available to the model when shuffling the original variable.
When set to `false`, no feature importance will be calculated.
Values must be one of the following:
* `True`
* `False`
* `native`
* `permutation`
* `null`
Toggle encoding of feature columns.
When enabled, Graphext will auto-convert any column types to the numeric type before
fitting the model. How this conversion is done can be configured using the `feature_encoder`
option below.
If disabled, any model trained in this step will assume that input data
is already in an appropriate format (e.g. numerical and not containing any missing values).
Configures encoding of feature columns.
By default (`null`), Graphext chooses automatically how to convert any column types the model
may not understand natively to a numeric type.
A configuration object can be passed instead to overwrite specific parameter values with respect
to their default values.
Numeric encoder.
Configures encoding of numeric features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Further parameters passed to the `scaler` function.
Details depend no the particular scaler used.
Boolean encoder.
Configures encoding of boolean features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Ordinal encoder.
Configures encoding of categorical features that have a natural order.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Category encoder.
May contain either a single configuration for all categorical variables, or two different configurations
for low- and high-cardinality variables. For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for categories with fewer than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Multilabel encoder.
Configures encoding of multivalued categorical features (variable length lists of categories,
or the semantic type `list[category]` for short). May contain either a single configuration for
all multilabel variables, or two different configurations for low- and high-cardinality variables.
For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for mulitabel columns with fewer than `cardinality_threshold` unique categories/labels.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Datetime encoder.
Configures encoding of datetime (timestamp) features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
A list of numerical components to extract.
Will create one numeric column for each component.
Each item in array.
Values must be one of the following:
`day` `dayofweek` `dayofyear` `hour` `minute` `month` `quarter` `season` `second` `week` `weekday` `weekofyear` `year`
A list of cyclical time features to extract.
"Cycles" are numerical transformations of features that should be represented on a circle. E.g. months,
ranging from 1 to 12, should be arranged such that 12 and 1 are next to each other, rather than on
opposite ends of a linear scale. We represent such cyclical time features on a circle by creating two
columns for each original feature: the sin and cos of the numerical feature after appropriate scaling.
Each item in array.
Values must be one of the following:
* `day`
* `dayofweek`
* `dayofyear`
* `hour`
* `month`
Whether to include the epoch as new feature (seconds since 01/01/1970).
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Embedding/vector encoder.
Configures encoding of multivalued numerical features (variable length lists of numbers, i.e. vectors, or the semantic type `list[number]` for short).
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Text encoder.
Configures encoding of text (natural language) features. Currently only allows
[Tf-Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) embeddings to represent texts. If you wish
to use other embeddings, e.g. semantic, Word2Vec etc., transform your text column first using
another step, and then use that result instead of the original texts.
Texts are *excluded* by default from the overall encoding of the dataset. See parameter
`include_text_features` below to active it.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Parameters to be passed to the text encoder (Tf-Idf parameters only for now).
See [scikit-learn's documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)
for detailed parameters and their explanation.
How many output features to generate.
The resulting Tf-Idf vectors will be reduced to these many dimensions (columns) using scikit-learn's
[truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
This performs a kind of [latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
By default we will reduce to 200 components.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_components ≤ 1024
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Whether to include or ignore text columns during the processing of input data.
Enabling this will convert texts to their TfIdf representation. Each text will be
converted to an N-dimensional vector in which each component measures the relative
"over-representation" of a specific word (or n-gram) relative to its overall
frequency in the whole dataset. This is disabled by default because it will
often be better to convert texts explicitly using a previous step, such as
`embed_text` or `embed_text_with_model`.
Model parameters.
Constant parameters to configure before training.
Regularization strengths to explore.
Smaller values specify stronger regularization. If `Cs` is as an integer, a grid of C values are chosen
in a logarithmic scale between 1e-4 and 1e4.
Values must be in the following range:
```javascript theme={null}
1 < Cs < inf
```
Each item in array.
Values must be in the following range:
```javascript theme={null}
0 ≤ Item < inf
```
Relative weights of l1 norm penalty vs l2 norm penalty to explore.
An l1-ratio of 0 means l2 penalty only (euclidean norm), resulting in a ridge regression penalizing large
coefficients proportional to their sum of squares. An l1-ratio of 1.0 means l1 penalty only (taxicab/manhattan norm),
i.e. proportional to the sum of absolute coefficient values. This has the tendency to prefer solutions with
fewer non-zero coefficients, effectively reducing the number of features used in the optimized model.
Each item in array.
Values must be in the following range:
```javascript theme={null}
0 ≤ Item ≤ 1
```
Maximum number of iterations of the optimization algorithm.
Try increasing this if you suspect the algorithm doesn't reach the peformance you'd expect.
Values must be in the following range:
```javascript theme={null}
100 ≤ max_iter < inf
```
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `roc_auc` `roc_auc_ovr` `roc_auc_ovo` `roc_auc_ovr_weighted` `roc_auc_ovo_weighted`
Mode for selecting sample weights given its class.
If not provided (or `null`), all weights will be 1, and so in effect no weights are applied.
When `"balanced"` (default), sample weights are calculated as inversely proportional to class
frequencies, such that samples from the more frequent classes have lower weights, and under-represented
classes are given more weight.
Values must be one of the following:
* `balanced`
Configure model validation.
Allows evaluation of model performance via [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html)
using custom metrics. If not specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_splits < inf
```
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [k-fold cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#k-fold)
to split the dataset. E.g. if `n_splits` is 5, the dataset will be split into 5 equal-sized parts.
For five iterations four parts will then be used for training and the remaining part for testing.
If `test_size` is a number between 0 and 1, in contrast, validation is done using a
[shuffle-split](https://scikit-learn.org/stable/modules/cross_validation.html#shufflesplit)
approach. Here, instead of splitting the data into `n_splits` equal parts up front, in each iteration
we randomize the data and sample a proportion equal to `test_size` to use for evaluation and the remaining
rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
Whether to split the data by time.
Most recent data will be used for testing and previous data for training. Assumes data is passed
already sorted ascending by time.
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `roc_auc` `roc_auc_ovr` `roc_auc_ovo` `roc_auc_ovr_weighted` `roc_auc_ovo_weighted`
Seed for random number generator ensuring reproducibility.
Values must be in the following range:
```javascript theme={null}
0 ≤ seed < inf
```
Sort the data before training.
If the data is not already sorted by time, you can sort it here. This is useful when you want to split the data
by time, for example to train on older data and test on newer data (see the `time_split` parameter in validation
configurations). If the data is already sorted by time, you can ignore this parameter.
One or more column to sort by.
Each item in array.
Sort order.
Whether to sort in ascending or descending order. If the single value `true` is provided, or no value is specified,
all columns will be sorted in ascending order. If a single `false` is provided, all columns will be sorted in descending
order. If an array of booleans is provided, each column will be sorted according to the corresponding boolean value.
Each item in array.
Trains a decision tree classifier.
A [decision tree](https://en.wikipedia.org/wiki/Decision_tree_learning) is a non-parametric,
supervised method for predicting a target variable by learning simple decision rules inferred
from the data. It can be seen as a piecewise constant approximation, applying simple if-else decision
rules to the data. The particular model used here is scikit-learn's
[DecisionTreeClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html).
Target variable (labels).
Name of the column that contains your target values (labels).
Name of the positive class.
In *binary* classification, usually the class you're most interested in, for example the label/class
corresponding to successful lead conversion in a lead score model, the class corresponding to a
customer who has churned in a churn prediction model, etc.
If provided, will automaticall measure the performance (accuracy, precision, recall) of the model on this
class, in addition to averages across all classes. If not provided, only summary metrics will be reported.
Maximum number of classes in the target variable.
If there are more classes than this, the least frequent classes will be grouped together into a single class
called "others". Reducing the number of classes in the target variable can help improve model performance,
especially when the number of classes is very large, some classes are very rare, or the dataset doesn't have
sufficient samples for all classes. Raising this significantly might lead to much longer training times.
Values must be in the following range:
```javascript theme={null}
2 ≤ max_classes ≤ 100
```
Importance of each feature in the model.
Whether and how to measure each feature's contribution to the model's predictions. The higher the value,
the more important the feature was in the model. Only relative values are meaningful, i.e. the importance
of a feature relative to other features in the model.
Also note that feature importance is usually meaningful only for models that fit the data well.
The default (`null`, `true` or `"native"`) uses the classifier's native feature importance measure, e.g.
[prediction-value-change](https://catboost.ai/en/docs/concepts/fstr#regular-feature-importance) in the case
of Catboost, [Gini importance](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier.feature_importances_)
in the case of scikit-learn's DecisionTreeClassifier, and the mean of absolute coefficients in the case of
logistic regression.
When set to `"permutation"`, uses [permutation importance](https://scikit-learn.org/stable/modules/permutation_importance.html),
i.e. measures the decrease in model score when a single feature's values are randomly shuffled. This is
considerably slower than native feature importance (the model needs to be evaluated an additional k\*n times,
where k is the number of features and n the number of repetitions to average over). On the positive side it is
model-agnostic and doesn't suffer from bias towards high cardinality features (like some tree-based feature
importances). On the negative side, it can be sensitive to strongly correlated features, as the unshuffled
correlated variable is still available to the model when shuffling the original variable.
When set to `false`, no feature importance will be calculated.
Values must be one of the following:
* `True`
* `False`
* `native`
* `permutation`
* `null`
Toggle encoding of feature columns.
When enabled, Graphext will auto-convert any column types to the numeric type before
fitting the model. How this conversion is done can be configured using the `feature_encoder`
option below.
If disabled, any model trained in this step will assume that input data
is already in an appropriate format (e.g. numerical and not containing any missing values).
Configures encoding of feature columns.
By default (`null`), Graphext chooses automatically how to convert any column types the model
may not understand natively to a numeric type.
A configuration object can be passed instead to overwrite specific parameter values with respect
to their default values.
Numeric encoder.
Configures encoding of numeric features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Further parameters passed to the `scaler` function.
Details depend no the particular scaler used.
Boolean encoder.
Configures encoding of boolean features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Ordinal encoder.
Configures encoding of categorical features that have a natural order.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Category encoder.
May contain either a single configuration for all categorical variables, or two different configurations
for low- and high-cardinality variables. For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for categories with fewer than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Multilabel encoder.
Configures encoding of multivalued categorical features (variable length lists of categories,
or the semantic type `list[category]` for short). May contain either a single configuration for
all multilabel variables, or two different configurations for low- and high-cardinality variables.
For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for mulitabel columns with fewer than `cardinality_threshold` unique categories/labels.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Datetime encoder.
Configures encoding of datetime (timestamp) features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
A list of numerical components to extract.
Will create one numeric column for each component.
Each item in array.
Values must be one of the following:
`day` `dayofweek` `dayofyear` `hour` `minute` `month` `quarter` `season` `second` `week` `weekday` `weekofyear` `year`
A list of cyclical time features to extract.
"Cycles" are numerical transformations of features that should be represented on a circle. E.g. months,
ranging from 1 to 12, should be arranged such that 12 and 1 are next to each other, rather than on
opposite ends of a linear scale. We represent such cyclical time features on a circle by creating two
columns for each original feature: the sin and cos of the numerical feature after appropriate scaling.
Each item in array.
Values must be one of the following:
* `day`
* `dayofweek`
* `dayofyear`
* `hour`
* `month`
Whether to include the epoch as new feature (seconds since 01/01/1970).
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Embedding/vector encoder.
Configures encoding of multivalued numerical features (variable length lists of numbers, i.e. vectors, or the semantic type `list[number]` for short).
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Text encoder.
Configures encoding of text (natural language) features. Currently only allows
[Tf-Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) embeddings to represent texts. If you wish
to use other embeddings, e.g. semantic, Word2Vec etc., transform your text column first using
another step, and then use that result instead of the original texts.
Texts are *excluded* by default from the overall encoding of the dataset. See parameter
`include_text_features` below to active it.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Parameters to be passed to the text encoder (Tf-Idf parameters only for now).
See [scikit-learn's documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)
for detailed parameters and their explanation.
How many output features to generate.
The resulting Tf-Idf vectors will be reduced to these many dimensions (columns) using scikit-learn's
[truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
This performs a kind of [latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
By default we will reduce to 200 components.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_components ≤ 1024
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Whether to include or ignore text columns during the processing of input data.
Enabling this will convert texts to their TfIdf representation. Each text will be
converted to an N-dimensional vector in which each component measures the relative
"over-representation" of a specific word (or n-gram) relative to its overall
frequency in the whole dataset. This is disabled by default because it will
often be better to convert texts explicitly using a previous step, such as
`embed_text` or `embed_text_with_model`.
Decision tree configuration parameters.
These parameters are specific to the decision tree algorithm. They are used to define the tree structure
and the stopping criteria. The default values are the ones used by scikit-learn.
For more information, see the [scikit-learn documentation](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier).
Function to measure the quality of a split.
Supported criteria are "gini" for the Gini impurity and "entropy" for the information gain.
Values must be one of the following:
* `gini`
* `entropy`
Strategy used to choose the split at each node.
Supported strategies are "best" to choose the best split and "random" to choose the best random split.
Values must be one of the following:
* `best`
* `random`
Maximum depth of the tree.
The maximum depth of the tree. If null, then nodes are expanded until all leaves are pure
or until all leaves contain less than min\_samples\_split samples.
Values must be in the following range:
```javascript theme={null}
1 < max_depth < inf
```
Minimum number of samples required to split an internal node.
The minimum number of samples required to split an internal node. If int, then consider min\_samples\_split
as the minimum *count*. If float, then min\_samples\_split is a *fraction* and `ceil(min_samples_split * n_samples)`
are the minimum number of samples for each split.
Minimum number of samples required to be at a leaf node.
The minimum number of samples required to be at a leaf node. A split point at any depth will only be
considered if it leaves at least `min_samples_leaf` training samples in each of the left and right branches.
This may have the effect of smoothing the model, especially in regression.
If int, then consider `min_samples_leaf` as the minimum *count*. If float, then `min_samples_leaf` is
a *fraction* and `ceil(min_samples_leaf * n_samples)` are the minimum number of samples for each node.
Grow a tree with `max_leaf_nodes` in best-first fashion.
Best nodes are defined as relative reduction in impurity. If null then unlimited number of leaf nodes.
Number of features to consider when looking for the best split.
The number of features to consider when looking for the best split:
* If int, then consider `max_features` features at each split.
* If float, then `max_features` is a *fraction* and `int(max_features * n_features)` features are considered at each split.
* If "auto", then `max_features=sqrt(n_features)`.
* If "sqrt", then `max_features=sqrt(n_features)`.
* If "log2", then `max_features=log2(n_features)`.
* If null, then `max_features=n_features`.
Note: the search for a split does not stop until at least one valid partition of the node samples is found,
even if it requires to effectively inspect more than `max_features` features.
Complexity parameter used for Minimal Cost-Complexity Pruning.
Minimal Cost-Complexity Pruning recursively finds the node with the "weakest link". The weakest link is
characterized by an effective alpha, where the nodes with the smallest effective alpha are pruned first.
As alpha increases, more of the tree is pruned, which increases the total impurity of its leaves.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ ccp_alpha < inf
```
Controls the randomness of the estimator.
The features are always randomly permuted at each split, even if splitter is set to "best". When
max\_features \< n\_features, the algorithm will select max\_features at random at each split before
finding the best split among them. But the best found split may vary across different runs, even if
max\_features=n\_features. That is the case, if the improvement of the criterion is identical for several
splits and one split has to be selected at random. To obtain a deterministic behaviour during fitting,
random\_state has to be fixed to an integer.
How to weigh each class of the target variable.
If `null`, all classes will have a weight of one. The "balanced" mode uses the values of the target y to
automatically adjust weights inversely proportional to class frequencies in the input data
as `n_samples / (n_classes * np.bincount(y))`.
Values must be one of the following:
* `balanced`
Configure model validation.
Allows evaluation of model performance via [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html)
using custom metrics. If not specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_splits < inf
```
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [k-fold cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#k-fold)
to split the dataset. E.g. if `n_splits` is 5, the dataset will be split into 5 equal-sized parts.
For five iterations four parts will then be used for training and the remaining part for testing.
If `test_size` is a number between 0 and 1, in contrast, validation is done using a
[shuffle-split](https://scikit-learn.org/stable/modules/cross_validation.html#shufflesplit)
approach. Here, instead of splitting the data into `n_splits` equal parts up front, in each iteration
we randomize the data and sample a proportion equal to `test_size` to use for evaluation and the remaining
rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
Whether to split the data by time.
Most recent data will be used for testing and previous data for training. Assumes data is passed
already sorted ascending by time.
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `roc_auc` `roc_auc_ovr` `roc_auc_ovo` `roc_auc_ovr_weighted` `roc_auc_ovo_weighted`
Configure hypertuning.
Configures the optimization of model hyper-parameters via cross-validated grid- or randomized search.
Which search strategy to use for optimization.
Grid search explores all possible combinations of parameters specified in `params`.
Randomized search, on the other hand, randomly samples `iterations` parameter combinations
from the distributions specified in `params`.
Values must be one of the following:
* `grid`
* `random`
How many randomly sampled parameter combinations to test in randomized search.
Values must be in the following range:
```javascript theme={null}
1 < iterations < inf
```
Configure model validation.
Allows evaluation of model performance via [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html)
using custom metrics. If not specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_splits < inf
```
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [k-fold cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#k-fold)
to split the dataset. E.g. if `n_splits` is 5, the dataset will be split into 5 equal-sized parts.
For five iterations four parts will then be used for training and the remaining part for testing.
If `test_size` is a number between 0 and 1, in contrast, validation is done using a
[shuffle-split](https://scikit-learn.org/stable/modules/cross_validation.html#shufflesplit)
approach. Here, instead of splitting the data into `n_splits` equal parts up front, in each iteration
we randomize the data and sample a proportion equal to `test_size` to use for evaluation and the remaining
rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
Whether to split the data by time.
Most recent data will be used for testing and previous data for training. Assumes data is passed
already sorted ascending by time.
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `roc_auc` `roc_auc_ovr` `roc_auc_ovo` `roc_auc_ovr_weighted` `roc_auc_ovo_weighted`
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `roc_auc` `roc_auc_ovr` `roc_auc_ovo` `roc_auc_ovr_weighted` `roc_auc_ovo_weighted`
The parameter values to explore.
Allows tuning of any and all of the parameters that can be set also as constants in the
"params" attribute.
Keys in this object should be strings identifying parameter names, and values should be
*lists* of values to explore for that parameter. E.g. `"depth": [3, 5, 7]`.
List of depths values to explore.
The maximum depth of the tree.
Values must be in the following range:
```javascript theme={null}
2 ≤ Item ≤ 16
```
List of iterations values to explore.
Number of iterations.
The maximum number of trees that can be built when solving machine learning problems. When using other parameters that limit the number of iterations, the final number of trees may be less than the number specified in this parameter.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item < inf
```
List of values configuring max cardinality for one-hot encoding.
Maximum cardinality of variables to be one-hot encoded.
Use one-hot encoding for all categorical features with a number of different values less than or equal to this value. Other variables will be target-encoded. Note that one-hot encoding is faster than the alternatives, so decreasing this value makes it more likely slower methods will be used. See [CatBoost details](https://catboost.ai/docs/concepts/algorithm-main-stages_cat-to-numberic.html) for further information.
Values must be in the following range:
```javascript theme={null}
2 ≤ Item < inf
```
List of values configuring variable combination complexity.
The maximum number of features that can be combined when transforming categorical variables.
Each resulting combination consists of one or more categorical features and can optionally contain binary features in the following form: “numeric feature > value”.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item ≤ 4
```
List of leaf regularization strengths.
Coefficient at the L2 regularization term of the cost function.
Values must be in the following range:
```javascript theme={null}
0.0 < Item < inf
```
List of border counts.
The number of splits for numerical features.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item ≤ 65535
```
List of random strengths.
The amount of randomness to use for scoring splits.
Use this parameter to avoid overfitting the model. The value multiplies the variance of a random variable (with
zero mean) that is added to the score used to select splits when a tree is grown.
Values must be in the following range:
```javascript theme={null}
0 < Item < inf
```
List of criterion values to explore.
Function to measure the quality of a split.
Supported criteria are "gini" for the Gini impurity and "entropy" for the information gain.
Values must be one of the following:
* `gini`
* `entropy`
List of splitter values to explore.
Strategy used to choose the split at each node.
Supported strategies are "best" to choose the best split and "random" to choose the best random split.
Values must be one of the following:
* `best`
* `random`
List of max\_depth values to explore.
Maximum depth of the tree.
The maximum depth of the tree. If null, then nodes are expanded until all leaves are pure
or until all leaves contain less than min\_samples\_split samples.
Values must be in the following range:
```javascript theme={null}
1 < Item < inf
```
List of min\_samples\_split values to explore.
Minimum number of samples required to split an internal node.
The minimum number of samples required to split an internal node. If int, then consider min\_samples\_split
as the minimum *count*. If float, then min\_samples\_split is a *fraction* and `ceil(min_samples_split * n_samples)`
are the minimum number of samples for each split.
List of min\_samples\_leaf values to explore.
Minimum number of samples required to be at a leaf node.
The minimum number of samples required to be at a leaf node. A split point at any depth will only be
considered if it leaves at least `min_samples_leaf` training samples in each of the left and right branches.
This may have the effect of smoothing the model, especially in regression.
If int, then consider `min_samples_leaf` as the minimum *count*. If float, then `min_samples_leaf` is
a *fraction* and `ceil(min_samples_leaf * n_samples)` are the minimum number of samples for each node.
List of max\_leaf\_nodes values to explore.
Grow a tree with `max_leaf_nodes` in best-first fashion.
Best nodes are defined as relative reduction in impurity. If null then unlimited number of leaf nodes.
List of max\_features values to explore.
Number of features to consider when looking for the best split.
The number of features to consider when looking for the best split:
* If int, then consider `max_features` features at each split.
* If float, then `max_features` is a *fraction* and `int(max_features * n_features)` features are considered at each split.
* If "auto", then `max_features=sqrt(n_features)`.
* If "sqrt", then `max_features=sqrt(n_features)`.
* If "log2", then `max_features=log2(n_features)`.
* If null, then `max_features=n_features`.
Note: the search for a split does not stop until at least one valid partition of the node samples is found,
even if it requires to effectively inspect more than `max_features` features.
List of ccp\_alpha values to explore.
Complexity parameter used for Minimal Cost-Complexity Pruning.
Minimal Cost-Complexity Pruning recursively finds the node with the "weakest link". The weakest link is
characterized by an effective alpha, where the nodes with the smallest effective alpha are pruned first.
As alpha increases, more of the tree is pruned, which increases the total impurity of its leaves.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ Item < inf
```
Seed for random number generator ensuring reproducibility.
Values must be in the following range:
```javascript theme={null}
0 ≤ seed < inf
```
Sort the data before training.
If the data is not already sorted by time, you can sort it here. This is useful when you want to split the data
by time, for example to train on older data and test on newer data (see the `time_split` parameter in validation
configurations). If the data is already sorted by time, you can ignore this parameter.
One or more column to sort by.
Each item in array.
Sort order.
Whether to sort in ascending or descending order. If the single value `true` is provided, or no value is specified,
all columns will be sorted in ascending order. If a single `false` is provided, all columns will be sorted in descending
order. If an array of booleans is provided, each column will be sorted according to the corresponding boolean value.
Each item in array.
Train a [RandomForest Classifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html).
Ensemble method using multiple decision trees for improved accuracy and over-fitting control.
A versatile ensemble learning method that constructs multiple decision trees during training and outputs the mode of the classes (classification)
or mean prediction (regression) of the individual trees to improve predictive accuracy and control over-fitting.
It operates by building a multitude of decision trees at training time and outputting the class that is the mode of the classes (classification)
or mean prediction (regression) of the individual trees. RandomForestClassifier is an ensemble of decision trees.
For classification tasks, the output of the RandomForestClassifier is the class selected by most trees.
It works by randomly selecting subsets of the training data, fitting a decision tree to each, and aggregating the predictions.
This process helps in reducing variance and avoids overfitting.
Target variable (labels).
Name of the column that contains your target values (labels).
Name of the positive class.
In *binary* classification, usually the class you're most interested in, for example the label/class
corresponding to successful lead conversion in a lead score model, the class corresponding to a
customer who has churned in a churn prediction model, etc.
If provided, will automaticall measure the performance (accuracy, precision, recall) of the model on this
class, in addition to averages across all classes. If not provided, only summary metrics will be reported.
Maximum number of classes in the target variable.
If there are more classes than this, the least frequent classes will be grouped together into a single class
called "others". Reducing the number of classes in the target variable can help improve model performance,
especially when the number of classes is very large, some classes are very rare, or the dataset doesn't have
sufficient samples for all classes. Raising this significantly might lead to much longer training times.
Values must be in the following range:
```javascript theme={null}
2 ≤ max_classes ≤ 100
```
Importance of each feature in the model.
Whether and how to measure each feature's contribution to the model's predictions. The higher the value,
the more important the feature was in the model. Only relative values are meaningful, i.e. the importance
of a feature relative to other features in the model.
Also note that feature importance is usually meaningful only for models that fit the data well.
The default (`null`, `true` or `"native"`) uses the classifier's native feature importance measure, e.g.
[prediction-value-change](https://catboost.ai/en/docs/concepts/fstr#regular-feature-importance) in the case
of Catboost, [Gini importance](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier.feature_importances_)
in the case of scikit-learn's DecisionTreeClassifier, and the mean of absolute coefficients in the case of
logistic regression.
When set to `"permutation"`, uses [permutation importance](https://scikit-learn.org/stable/modules/permutation_importance.html),
i.e. measures the decrease in model score when a single feature's values are randomly shuffled. This is
considerably slower than native feature importance (the model needs to be evaluated an additional k\*n times,
where k is the number of features and n the number of repetitions to average over). On the positive side it is
model-agnostic and doesn't suffer from bias towards high cardinality features (like some tree-based feature
importances). On the negative side, it can be sensitive to strongly correlated features, as the unshuffled
correlated variable is still available to the model when shuffling the original variable.
When set to `false`, no feature importance will be calculated.
Values must be one of the following:
* `True`
* `False`
* `native`
* `permutation`
* `null`
Toggle encoding of feature columns.
When enabled, Graphext will auto-convert any column types to the numeric type before
fitting the model. How this conversion is done can be configured using the `feature_encoder`
option below.
If disabled, any model trained in this step will assume that input data
is already in an appropriate format (e.g. numerical and not containing any missing values).
Configures encoding of feature columns.
By default (`null`), Graphext chooses automatically how to convert any column types the model
may not understand natively to a numeric type.
A configuration object can be passed instead to overwrite specific parameter values with respect
to their default values.
Numeric encoder.
Configures encoding of numeric features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Further parameters passed to the `scaler` function.
Details depend no the particular scaler used.
Boolean encoder.
Configures encoding of boolean features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Ordinal encoder.
Configures encoding of categorical features that have a natural order.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Category encoder.
May contain either a single configuration for all categorical variables, or two different configurations
for low- and high-cardinality variables. For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for categories with fewer than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Multilabel encoder.
Configures encoding of multivalued categorical features (variable length lists of categories,
or the semantic type `list[category]` for short). May contain either a single configuration for
all multilabel variables, or two different configurations for low- and high-cardinality variables.
For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for mulitabel columns with fewer than `cardinality_threshold` unique categories/labels.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Datetime encoder.
Configures encoding of datetime (timestamp) features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
A list of numerical components to extract.
Will create one numeric column for each component.
Each item in array.
Values must be one of the following:
`day` `dayofweek` `dayofyear` `hour` `minute` `month` `quarter` `season` `second` `week` `weekday` `weekofyear` `year`
A list of cyclical time features to extract.
"Cycles" are numerical transformations of features that should be represented on a circle. E.g. months,
ranging from 1 to 12, should be arranged such that 12 and 1 are next to each other, rather than on
opposite ends of a linear scale. We represent such cyclical time features on a circle by creating two
columns for each original feature: the sin and cos of the numerical feature after appropriate scaling.
Each item in array.
Values must be one of the following:
* `day`
* `dayofweek`
* `dayofyear`
* `hour`
* `month`
Whether to include the epoch as new feature (seconds since 01/01/1970).
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Embedding/vector encoder.
Configures encoding of multivalued numerical features (variable length lists of numbers, i.e. vectors, or the semantic type `list[number]` for short).
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Text encoder.
Configures encoding of text (natural language) features. Currently only allows
[Tf-Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) embeddings to represent texts. If you wish
to use other embeddings, e.g. semantic, Word2Vec etc., transform your text column first using
another step, and then use that result instead of the original texts.
Texts are *excluded* by default from the overall encoding of the dataset. See parameter
`include_text_features` below to active it.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Parameters to be passed to the text encoder (Tf-Idf parameters only for now).
See [scikit-learn's documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)
for detailed parameters and their explanation.
How many output features to generate.
The resulting Tf-Idf vectors will be reduced to these many dimensions (columns) using scikit-learn's
[truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
This performs a kind of [latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
By default we will reduce to 200 components.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_components ≤ 1024
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Whether to include or ignore text columns during the processing of input data.
Enabling this will convert texts to their TfIdf representation. Each text will be
converted to an N-dimensional vector in which each component measures the relative
"over-representation" of a specific word (or n-gram) relative to its overall
frequency in the whole dataset. This is disabled by default because it will
often be better to convert texts explicitly using a previous step, such as
`embed_text` or `embed_text_with_model`.
RandomForest configuration parameters.
You can check the official documentation for more details about RandomForest's parameters [here](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html).
The number of trees in the forest.
The number of trees in the forest. A larger number of trees increases the performance but also the computational cost.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_estimators < inf
```
The function to measure the quality of a split.
The function to measure the quality of a split. Supported criteria are "gini" for the Gini impurity and "entropy" for the information gain.
Values must be one of the following:
* `gini`
* `entropy`
The maximum depth of the tree.
The maximum depth of the tree. If `null`, then nodes are expanded until all leaves are pure or until all leaves contain less than min\_samples\_split samples.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_depth < inf
```
How to weigh each class of the target variable.
If `null`, all classes will have a weight of one. The "balanced" mode uses the values of the target y to
automatically adjust weights inversely proportional to class frequencies in the input data
as `n_samples / (n_classes * np.bincount(y))`.
The “balanced\_subsample” mode is the same as “balanced” except that weights are computed based on the bootstrap
sample for every tree grown.
Values must be one of the following:
* `balanced`
* `balanced_subsample`
* `None`
The minimum number of samples required to split an internal node.
The minimum number of samples required to split an internal node. A split point at any depth will only be considered if it leaves at least `min_samples_split` training samples in each of the left and right branches.
Values must be in the following range:
```javascript theme={null}
2 ≤ min_samples_split < inf
```
The minimum number of samples required to be at a leaf node.
A split point at any depth will only be considered if it leaves at least `min_samples_leaf` training samples in each of the left and right branches. This may have the effect of smoothing the model.
Values must be in the following range:
```javascript theme={null}
1 ≤ min_samples_leaf < inf
```
The minimum weighted fraction of the sum total of weights required to be at a leaf node.
The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node. Samples have equal weight when sample\_weight is not provided.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ min_weight_fraction_leaf ≤ 0.5
```
The number of features to consider when looking for the best split.
The number of features to consider when looking for the best split. If “auto”, then `max_features=sqrt(n_features)`. If `null`, then `max_features=n_features`.
Grow trees with max\_leaf\_nodes in best-first fashion.
Grow trees with `max_leaf_nodes` in best-first fashion. Best nodes are defined as relative reduction in impurity. If `null` then unlimited number of leaf nodes.
A node will be split if this split induces a decrease of the impurity greater than or equal to this value.
A node will be split if this split induces a decrease of the impurity greater than or equal to this value. This may have the effect of smoothing the model, especially in regression.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ min_impurity_decrease < inf
```
Whether bootstrap samples are used when building trees.
If `true`, bootstrap samples are used when building trees. If `false`, the whole dataset is used to build each tree.
Controls both the randomness of the bootstrapping of the samples used when building trees and the sampling of the features to consider when looking for the best split at each node.
If bootstrap is True, the number of samples to draw from X to train each base estimator.
If bootstrap is True, the number of samples to draw from X to train each base estimator. If `null` (default), then draw `X.shape[0]` samples.
Complexity parameter used for Minimal Cost-Complexity Pruning.
Complexity parameter used for Minimal Cost-Complexity Pruning. The subtree with the largest cost complexity that is smaller than `ccp_alpha` will be chosen. By default, no pruning is performed.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ ccp_alpha < inf
```
The number of jobs to run in parallel for both `fit` and `predict`.
The number of jobs to run in parallel for both `fit` and `predict`. `-1` means using all processors.
Controls the verbosity when fitting and predicting.
Controls the verbosity when fitting and predicting.
Values must be in the following range:
```javascript theme={null}
0 ≤ verbose < inf
```
Configure model validation.
Allows evaluation of model performance via [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html)
using custom metrics. If not specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_splits < inf
```
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [k-fold cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#k-fold)
to split the dataset. E.g. if `n_splits` is 5, the dataset will be split into 5 equal-sized parts.
For five iterations four parts will then be used for training and the remaining part for testing.
If `test_size` is a number between 0 and 1, in contrast, validation is done using a
[shuffle-split](https://scikit-learn.org/stable/modules/cross_validation.html#shufflesplit)
approach. Here, instead of splitting the data into `n_splits` equal parts up front, in each iteration
we randomize the data and sample a proportion equal to `test_size` to use for evaluation and the remaining
rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
Whether to split the data by time.
Most recent data will be used for testing and previous data for training. Assumes data is passed
already sorted ascending by time.
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `roc_auc` `roc_auc_ovr` `roc_auc_ovo` `roc_auc_ovr_weighted` `roc_auc_ovo_weighted`
Configure hypertuning.
Configures the optimization of model hyper-parameters via cross-validated grid- or randomized search.
Which search strategy to use for optimization.
Grid search explores all possible combinations of parameters specified in `params`.
Randomized search, on the other hand, randomly samples `iterations` parameter combinations
from the distributions specified in `params`.
Values must be one of the following:
* `grid`
* `random`
How many randomly sampled parameter combinations to test in randomized search.
Values must be in the following range:
```javascript theme={null}
1 < iterations < inf
```
Configure model validation.
Allows evaluation of model performance via [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html)
using custom metrics. If not specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_splits < inf
```
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [k-fold cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#k-fold)
to split the dataset. E.g. if `n_splits` is 5, the dataset will be split into 5 equal-sized parts.
For five iterations four parts will then be used for training and the remaining part for testing.
If `test_size` is a number between 0 and 1, in contrast, validation is done using a
[shuffle-split](https://scikit-learn.org/stable/modules/cross_validation.html#shufflesplit)
approach. Here, instead of splitting the data into `n_splits` equal parts up front, in each iteration
we randomize the data and sample a proportion equal to `test_size` to use for evaluation and the remaining
rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
Whether to split the data by time.
Most recent data will be used for testing and previous data for training. Assumes data is passed
already sorted ascending by time.
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `roc_auc` `roc_auc_ovr` `roc_auc_ovo` `roc_auc_ovr_weighted` `roc_auc_ovo_weighted`
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `roc_auc` `roc_auc_ovr` `roc_auc_ovo` `roc_auc_ovr_weighted` `roc_auc_ovo_weighted`
The parameter values to explore.
Allows tuning of any and all of the parameters that can be set also as constants in the
"params" attribute.
Keys in this object should be strings identifying parameter names, and values should be
*lists* of values to explore for that parameter. E.g. `"depth": [3, 5, 7]`.
List of depths values to explore.
The maximum depth of the tree.
Values must be in the following range:
```javascript theme={null}
2 ≤ Item ≤ 16
```
List of iterations values to explore.
Number of iterations.
The maximum number of trees that can be built when solving machine learning problems. When using other parameters that limit the number of iterations, the final number of trees may be less than the number specified in this parameter.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item < inf
```
List of values configuring max cardinality for one-hot encoding.
Maximum cardinality of variables to be one-hot encoded.
Use one-hot encoding for all categorical features with a number of different values less than or equal to this value. Other variables will be target-encoded. Note that one-hot encoding is faster than the alternatives, so decreasing this value makes it more likely slower methods will be used. See [CatBoost details](https://catboost.ai/docs/concepts/algorithm-main-stages_cat-to-numberic.html) for further information.
Values must be in the following range:
```javascript theme={null}
2 ≤ Item < inf
```
List of values configuring variable combination complexity.
The maximum number of features that can be combined when transforming categorical variables.
Each resulting combination consists of one or more categorical features and can optionally contain binary features in the following form: “numeric feature > value”.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item ≤ 4
```
List of leaf regularization strengths.
Coefficient at the L2 regularization term of the cost function.
Values must be in the following range:
```javascript theme={null}
0.0 < Item < inf
```
List of border counts.
The number of splits for numerical features.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item ≤ 65535
```
List of random strengths.
The amount of randomness to use for scoring splits.
Use this parameter to avoid overfitting the model. The value multiplies the variance of a random variable (with
zero mean) that is added to the score used to select splits when a tree is grown.
Values must be in the following range:
```javascript theme={null}
0 < Item < inf
```
List of criterion values to explore.
Function to measure the quality of a split.
Supported criteria are "gini" for the Gini impurity and "entropy" for the information gain.
Values must be one of the following:
* `gini`
* `entropy`
List of splitter values to explore.
Strategy used to choose the split at each node.
Supported strategies are "best" to choose the best split and "random" to choose the best random split.
Values must be one of the following:
* `best`
* `random`
List of max\_depth values to explore.
Maximum depth of the tree.
The maximum depth of the tree. If null, then nodes are expanded until all leaves are pure
or until all leaves contain less than min\_samples\_split samples.
Values must be in the following range:
```javascript theme={null}
1 < Item < inf
```
List of min\_samples\_split values to explore.
Minimum number of samples required to split an internal node.
The minimum number of samples required to split an internal node. If int, then consider min\_samples\_split
as the minimum *count*. If float, then min\_samples\_split is a *fraction* and `ceil(min_samples_split * n_samples)`
are the minimum number of samples for each split.
List of min\_samples\_leaf values to explore.
Minimum number of samples required to be at a leaf node.
The minimum number of samples required to be at a leaf node. A split point at any depth will only be
considered if it leaves at least `min_samples_leaf` training samples in each of the left and right branches.
This may have the effect of smoothing the model, especially in regression.
If int, then consider `min_samples_leaf` as the minimum *count*. If float, then `min_samples_leaf` is
a *fraction* and `ceil(min_samples_leaf * n_samples)` are the minimum number of samples for each node.
List of max\_leaf\_nodes values to explore.
Grow a tree with `max_leaf_nodes` in best-first fashion.
Best nodes are defined as relative reduction in impurity. If null then unlimited number of leaf nodes.
List of max\_features values to explore.
Number of features to consider when looking for the best split.
The number of features to consider when looking for the best split:
* If int, then consider `max_features` features at each split.
* If float, then `max_features` is a *fraction* and `int(max_features * n_features)` features are considered at each split.
* If "auto", then `max_features=sqrt(n_features)`.
* If "sqrt", then `max_features=sqrt(n_features)`.
* If "log2", then `max_features=log2(n_features)`.
* If null, then `max_features=n_features`.
Note: the search for a split does not stop until at least one valid partition of the node samples is found,
even if it requires to effectively inspect more than `max_features` features.
List of ccp\_alpha values to explore.
Complexity parameter used for Minimal Cost-Complexity Pruning.
Minimal Cost-Complexity Pruning recursively finds the node with the "weakest link". The weakest link is
characterized by an effective alpha, where the nodes with the smallest effective alpha are pruned first.
As alpha increases, more of the tree is pruned, which increases the total impurity of its leaves.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ Item < inf
```
Seed for random number generator ensuring reproducibility.
Values must be in the following range:
```javascript theme={null}
0 ≤ seed < inf
```
Sort the data before training.
If the data is not already sorted by time, you can sort it here. This is useful when you want to split the data
by time, for example to train on older data and test on newer data (see the `time_split` parameter in validation
configurations). If the data is already sorted by time, you can ignore this parameter.
One or more column to sort by.
Each item in array.
Sort order.
Whether to sort in ascending or descending order. If the single value `true` is provided, or no value is specified,
all columns will be sorted in ascending order. If a single `false` is provided, all columns will be sorted in descending
order. If an array of booleans is provided, each column will be sorted according to the corresponding boolean value.
Each item in array.
Train an [ExtraTrees Classifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.ExtraTreesClassifier.html).
Fits a number of randomized decision trees for improved accuracy.
The ExtraTreesClassifier is an ensemble learning method fundamentally similar to a random forest.
It fits a number of randomized decision trees on various sub-samples of the dataset and uses
averaging to improve the predictive accuracy and control over-fitting.
The main difference from the random forest is in the way it splits nodes, which is random in ExtraTrees,
leading to more diversified trees and thus, a more robust model against overfitting on the training data.
Target variable (labels).
Name of the column that contains your target values (labels).
Name of the positive class.
In *binary* classification, usually the class you're most interested in, for example the label/class
corresponding to successful lead conversion in a lead score model, the class corresponding to a
customer who has churned in a churn prediction model, etc.
If provided, will automaticall measure the performance (accuracy, precision, recall) of the model on this
class, in addition to averages across all classes. If not provided, only summary metrics will be reported.
Maximum number of classes in the target variable.
If there are more classes than this, the least frequent classes will be grouped together into a single class
called "others". Reducing the number of classes in the target variable can help improve model performance,
especially when the number of classes is very large, some classes are very rare, or the dataset doesn't have
sufficient samples for all classes. Raising this significantly might lead to much longer training times.
Values must be in the following range:
```javascript theme={null}
2 ≤ max_classes ≤ 100
```
Importance of each feature in the model.
Whether and how to measure each feature's contribution to the model's predictions. The higher the value,
the more important the feature was in the model. Only relative values are meaningful, i.e. the importance
of a feature relative to other features in the model.
Also note that feature importance is usually meaningful only for models that fit the data well.
The default (`null`, `true` or `"native"`) uses the classifier's native feature importance measure, e.g.
[prediction-value-change](https://catboost.ai/en/docs/concepts/fstr#regular-feature-importance) in the case
of Catboost, [Gini importance](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier.feature_importances_)
in the case of scikit-learn's DecisionTreeClassifier, and the mean of absolute coefficients in the case of
logistic regression.
When set to `"permutation"`, uses [permutation importance](https://scikit-learn.org/stable/modules/permutation_importance.html),
i.e. measures the decrease in model score when a single feature's values are randomly shuffled. This is
considerably slower than native feature importance (the model needs to be evaluated an additional k\*n times,
where k is the number of features and n the number of repetitions to average over). On the positive side it is
model-agnostic and doesn't suffer from bias towards high cardinality features (like some tree-based feature
importances). On the negative side, it can be sensitive to strongly correlated features, as the unshuffled
correlated variable is still available to the model when shuffling the original variable.
When set to `false`, no feature importance will be calculated.
Values must be one of the following:
* `True`
* `False`
* `native`
* `permutation`
* `null`
Toggle encoding of feature columns.
When enabled, Graphext will auto-convert any column types to the numeric type before
fitting the model. How this conversion is done can be configured using the `feature_encoder`
option below.
If disabled, any model trained in this step will assume that input data
is already in an appropriate format (e.g. numerical and not containing any missing values).
Configures encoding of feature columns.
By default (`null`), Graphext chooses automatically how to convert any column types the model
may not understand natively to a numeric type.
A configuration object can be passed instead to overwrite specific parameter values with respect
to their default values.
Numeric encoder.
Configures encoding of numeric features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Further parameters passed to the `scaler` function.
Details depend no the particular scaler used.
Boolean encoder.
Configures encoding of boolean features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Ordinal encoder.
Configures encoding of categorical features that have a natural order.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Category encoder.
May contain either a single configuration for all categorical variables, or two different configurations
for low- and high-cardinality variables. For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for categories with fewer than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Multilabel encoder.
Configures encoding of multivalued categorical features (variable length lists of categories,
or the semantic type `list[category]` for short). May contain either a single configuration for
all multilabel variables, or two different configurations for low- and high-cardinality variables.
For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for mulitabel columns with fewer than `cardinality_threshold` unique categories/labels.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Datetime encoder.
Configures encoding of datetime (timestamp) features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
A list of numerical components to extract.
Will create one numeric column for each component.
Each item in array.
Values must be one of the following:
`day` `dayofweek` `dayofyear` `hour` `minute` `month` `quarter` `season` `second` `week` `weekday` `weekofyear` `year`
A list of cyclical time features to extract.
"Cycles" are numerical transformations of features that should be represented on a circle. E.g. months,
ranging from 1 to 12, should be arranged such that 12 and 1 are next to each other, rather than on
opposite ends of a linear scale. We represent such cyclical time features on a circle by creating two
columns for each original feature: the sin and cos of the numerical feature after appropriate scaling.
Each item in array.
Values must be one of the following:
* `day`
* `dayofweek`
* `dayofyear`
* `hour`
* `month`
Whether to include the epoch as new feature (seconds since 01/01/1970).
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Embedding/vector encoder.
Configures encoding of multivalued numerical features (variable length lists of numbers, i.e. vectors, or the semantic type `list[number]` for short).
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Text encoder.
Configures encoding of text (natural language) features. Currently only allows
[Tf-Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) embeddings to represent texts. If you wish
to use other embeddings, e.g. semantic, Word2Vec etc., transform your text column first using
another step, and then use that result instead of the original texts.
Texts are *excluded* by default from the overall encoding of the dataset. See parameter
`include_text_features` below to active it.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Parameters to be passed to the text encoder (Tf-Idf parameters only for now).
See [scikit-learn's documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)
for detailed parameters and their explanation.
How many output features to generate.
The resulting Tf-Idf vectors will be reduced to these many dimensions (columns) using scikit-learn's
[truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
This performs a kind of [latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
By default we will reduce to 200 components.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_components ≤ 1024
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Whether to include or ignore text columns during the processing of input data.
Enabling this will convert texts to their TfIdf representation. Each text will be
converted to an N-dimensional vector in which each component measures the relative
"over-representation" of a specific word (or n-gram) relative to its overall
frequency in the whole dataset. This is disabled by default because it will
often be better to convert texts explicitly using a previous step, such as
`embed_text` or `embed_text_with_model`.
RandomForest configuration parameters.
You can check the official documentation for more details about RandomForest's parameters [here](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html).
The number of trees in the forest.
The number of trees in the forest. A larger number of trees increases the performance but also the computational cost.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_estimators < inf
```
The function to measure the quality of a split.
The function to measure the quality of a split. Supported criteria are "gini" for the Gini impurity and "entropy" for the information gain.
Values must be one of the following:
* `gini`
* `entropy`
The maximum depth of the tree.
The maximum depth of the tree. If `null`, then nodes are expanded until all leaves are pure or until all leaves contain less than min\_samples\_split samples.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_depth < inf
```
How to weigh each class of the target variable.
If `null`, all classes will have a weight of one. The "balanced" mode uses the values of the target y to
automatically adjust weights inversely proportional to class frequencies in the input data
as `n_samples / (n_classes * np.bincount(y))`.
The “balanced\_subsample” mode is the same as “balanced” except that weights are computed based on the bootstrap
sample for every tree grown.
Values must be one of the following:
* `balanced`
* `balanced_subsample`
* `None`
The minimum number of samples required to split an internal node.
The minimum number of samples required to split an internal node. A split point at any depth will only be considered if it leaves at least `min_samples_split` training samples in each of the left and right branches.
Values must be in the following range:
```javascript theme={null}
2 ≤ min_samples_split < inf
```
The minimum number of samples required to be at a leaf node.
A split point at any depth will only be considered if it leaves at least `min_samples_leaf` training samples in each of the left and right branches. This may have the effect of smoothing the model.
Values must be in the following range:
```javascript theme={null}
1 ≤ min_samples_leaf < inf
```
The minimum weighted fraction of the sum total of weights required to be at a leaf node.
The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node. Samples have equal weight when sample\_weight is not provided.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ min_weight_fraction_leaf ≤ 0.5
```
The number of features to consider when looking for the best split.
The number of features to consider when looking for the best split. If “auto”, then `max_features=sqrt(n_features)`. If `null`, then `max_features=n_features`.
Grow trees with max\_leaf\_nodes in best-first fashion.
Grow trees with `max_leaf_nodes` in best-first fashion. Best nodes are defined as relative reduction in impurity. If `null` then unlimited number of leaf nodes.
A node will be split if this split induces a decrease of the impurity greater than or equal to this value.
A node will be split if this split induces a decrease of the impurity greater than or equal to this value. This may have the effect of smoothing the model, especially in regression.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ min_impurity_decrease < inf
```
Whether bootstrap samples are used when building trees.
If `true`, bootstrap samples are used when building trees. If `false`, the whole dataset is used to build each tree.
Controls both the randomness of the bootstrapping of the samples used when building trees and the sampling of the features to consider when looking for the best split at each node.
If bootstrap is True, the number of samples to draw from X to train each base estimator.
If bootstrap is True, the number of samples to draw from X to train each base estimator. If `null` (default), then draw `X.shape[0]` samples.
Complexity parameter used for Minimal Cost-Complexity Pruning.
Complexity parameter used for Minimal Cost-Complexity Pruning. The subtree with the largest cost complexity that is smaller than `ccp_alpha` will be chosen. By default, no pruning is performed.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ ccp_alpha < inf
```
The number of jobs to run in parallel for both `fit` and `predict`.
The number of jobs to run in parallel for both `fit` and `predict`. `-1` means using all processors.
Controls the verbosity when fitting and predicting.
Controls the verbosity when fitting and predicting.
Values must be in the following range:
```javascript theme={null}
0 ≤ verbose < inf
```
Configure model validation.
Allows evaluation of model performance via [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html)
using custom metrics. If not specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_splits < inf
```
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [k-fold cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#k-fold)
to split the dataset. E.g. if `n_splits` is 5, the dataset will be split into 5 equal-sized parts.
For five iterations four parts will then be used for training and the remaining part for testing.
If `test_size` is a number between 0 and 1, in contrast, validation is done using a
[shuffle-split](https://scikit-learn.org/stable/modules/cross_validation.html#shufflesplit)
approach. Here, instead of splitting the data into `n_splits` equal parts up front, in each iteration
we randomize the data and sample a proportion equal to `test_size` to use for evaluation and the remaining
rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
Whether to split the data by time.
Most recent data will be used for testing and previous data for training. Assumes data is passed
already sorted ascending by time.
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `roc_auc` `roc_auc_ovr` `roc_auc_ovo` `roc_auc_ovr_weighted` `roc_auc_ovo_weighted`
Configure hypertuning.
Configures the optimization of model hyper-parameters via cross-validated grid- or randomized search.
Which search strategy to use for optimization.
Grid search explores all possible combinations of parameters specified in `params`.
Randomized search, on the other hand, randomly samples `iterations` parameter combinations
from the distributions specified in `params`.
Values must be one of the following:
* `grid`
* `random`
How many randomly sampled parameter combinations to test in randomized search.
Values must be in the following range:
```javascript theme={null}
1 < iterations < inf
```
Configure model validation.
Allows evaluation of model performance via [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html)
using custom metrics. If not specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_splits < inf
```
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [k-fold cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#k-fold)
to split the dataset. E.g. if `n_splits` is 5, the dataset will be split into 5 equal-sized parts.
For five iterations four parts will then be used for training and the remaining part for testing.
If `test_size` is a number between 0 and 1, in contrast, validation is done using a
[shuffle-split](https://scikit-learn.org/stable/modules/cross_validation.html#shufflesplit)
approach. Here, instead of splitting the data into `n_splits` equal parts up front, in each iteration
we randomize the data and sample a proportion equal to `test_size` to use for evaluation and the remaining
rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
Whether to split the data by time.
Most recent data will be used for testing and previous data for training. Assumes data is passed
already sorted ascending by time.
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `roc_auc` `roc_auc_ovr` `roc_auc_ovo` `roc_auc_ovr_weighted` `roc_auc_ovo_weighted`
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `roc_auc` `roc_auc_ovr` `roc_auc_ovo` `roc_auc_ovr_weighted` `roc_auc_ovo_weighted`
The parameter values to explore.
Allows tuning of any and all of the parameters that can be set also as constants in the
"params" attribute.
Keys in this object should be strings identifying parameter names, and values should be
*lists* of values to explore for that parameter. E.g. `"depth": [3, 5, 7]`.
List of depths values to explore.
The maximum depth of the tree.
Values must be in the following range:
```javascript theme={null}
2 ≤ Item ≤ 16
```
List of iterations values to explore.
Number of iterations.
The maximum number of trees that can be built when solving machine learning problems. When using other parameters that limit the number of iterations, the final number of trees may be less than the number specified in this parameter.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item < inf
```
List of values configuring max cardinality for one-hot encoding.
Maximum cardinality of variables to be one-hot encoded.
Use one-hot encoding for all categorical features with a number of different values less than or equal to this value. Other variables will be target-encoded. Note that one-hot encoding is faster than the alternatives, so decreasing this value makes it more likely slower methods will be used. See [CatBoost details](https://catboost.ai/docs/concepts/algorithm-main-stages_cat-to-numberic.html) for further information.
Values must be in the following range:
```javascript theme={null}
2 ≤ Item < inf
```
List of values configuring variable combination complexity.
The maximum number of features that can be combined when transforming categorical variables.
Each resulting combination consists of one or more categorical features and can optionally contain binary features in the following form: “numeric feature > value”.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item ≤ 4
```
List of leaf regularization strengths.
Coefficient at the L2 regularization term of the cost function.
Values must be in the following range:
```javascript theme={null}
0.0 < Item < inf
```
List of border counts.
The number of splits for numerical features.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item ≤ 65535
```
List of random strengths.
The amount of randomness to use for scoring splits.
Use this parameter to avoid overfitting the model. The value multiplies the variance of a random variable (with
zero mean) that is added to the score used to select splits when a tree is grown.
Values must be in the following range:
```javascript theme={null}
0 < Item < inf
```
List of criterion values to explore.
Function to measure the quality of a split.
Supported criteria are "gini" for the Gini impurity and "entropy" for the information gain.
Values must be one of the following:
* `gini`
* `entropy`
List of splitter values to explore.
Strategy used to choose the split at each node.
Supported strategies are "best" to choose the best split and "random" to choose the best random split.
Values must be one of the following:
* `best`
* `random`
List of max\_depth values to explore.
Maximum depth of the tree.
The maximum depth of the tree. If null, then nodes are expanded until all leaves are pure
or until all leaves contain less than min\_samples\_split samples.
Values must be in the following range:
```javascript theme={null}
1 < Item < inf
```
List of min\_samples\_split values to explore.
Minimum number of samples required to split an internal node.
The minimum number of samples required to split an internal node. If int, then consider min\_samples\_split
as the minimum *count*. If float, then min\_samples\_split is a *fraction* and `ceil(min_samples_split * n_samples)`
are the minimum number of samples for each split.
List of min\_samples\_leaf values to explore.
Minimum number of samples required to be at a leaf node.
The minimum number of samples required to be at a leaf node. A split point at any depth will only be
considered if it leaves at least `min_samples_leaf` training samples in each of the left and right branches.
This may have the effect of smoothing the model, especially in regression.
If int, then consider `min_samples_leaf` as the minimum *count*. If float, then `min_samples_leaf` is
a *fraction* and `ceil(min_samples_leaf * n_samples)` are the minimum number of samples for each node.
List of max\_leaf\_nodes values to explore.
Grow a tree with `max_leaf_nodes` in best-first fashion.
Best nodes are defined as relative reduction in impurity. If null then unlimited number of leaf nodes.
List of max\_features values to explore.
Number of features to consider when looking for the best split.
The number of features to consider when looking for the best split:
* If int, then consider `max_features` features at each split.
* If float, then `max_features` is a *fraction* and `int(max_features * n_features)` features are considered at each split.
* If "auto", then `max_features=sqrt(n_features)`.
* If "sqrt", then `max_features=sqrt(n_features)`.
* If "log2", then `max_features=log2(n_features)`.
* If null, then `max_features=n_features`.
Note: the search for a split does not stop until at least one valid partition of the node samples is found,
even if it requires to effectively inspect more than `max_features` features.
List of ccp\_alpha values to explore.
Complexity parameter used for Minimal Cost-Complexity Pruning.
Minimal Cost-Complexity Pruning recursively finds the node with the "weakest link". The weakest link is
characterized by an effective alpha, where the nodes with the smallest effective alpha are pruned first.
As alpha increases, more of the tree is pruned, which increases the total impurity of its leaves.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ Item < inf
```
Seed for random number generator ensuring reproducibility.
Values must be in the following range:
```javascript theme={null}
0 ≤ seed < inf
```
Sort the data before training.
If the data is not already sorted by time, you can sort it here. This is useful when you want to split the data
by time, for example to train on older data and test on newer data (see the `time_split` parameter in validation
configurations). If the data is already sorted by time, you can ignore this parameter.
One or more column to sort by.
Each item in array.
Sort order.
Whether to sort in ascending or descending order. If the single value `true` is provided, or no value is specified,
all columns will be sorted in ascending order. If a single `false` is provided, all columns will be sorted in descending
order. If an array of booleans is provided, each column will be sorted according to the corresponding boolean value.
Each item in array.
# train_classification_gpu
Source: https://docs.graphext.com/api-docs/analyse/train_and_predict/train_classification_gpu
Train and store a classification model to be loaded at a later point for prediction.
The output will consist of a new column with the trained model's predictions on the training data,
as well as a saved and named model file that can be used in other projects for prediction of new data.
Optionally, if a second output column name is provided, the model's predicted probabilities will also be
returned.
A detailed guide on how to configure this step for model tuning and performance evaluation can be found
[here](/guides/model_train_eval/).
## Usage
The following examples show how the step can be used in a recipe.
Train a classification model with default parameters. By default, a Catboost model will be trained, but this can be changed to any of the supported models by specifying the `model` parameter (see below for details):
```stan theme={null}
train_classification(ds, {"target": "class"}) -> (ds.predicted, model)
```
To also return the predicted probabilities, provide a second column name:
```stan theme={null}
train_classification(ds, {"target": "class"}) -> (ds.predicted, ds.probs, model)
```
To be more explicit about which model parameters to use during training, which parameters to optimize (tune) automatically, and how to evaluate the model's performance, the following example shows a complete configuration. It will explicitly select the CatboostClassifier as the model, set `boosting_type` to "ordered", and select the best combination of `learning_rate` and `depth` from the values specified in the `tune: params` configuration. To find the best parameters, it will perform 5-fold cross-validation on each combination, and will use the scorer `f1_weighted` to measure the performance. `accuracy` will also be measured, but only for the purpose of reporting. The best parameter combination will than be evaluated on a single split of the dataset (with 20% of rows used for testing and 80% for training), with metrics selected automatically. Note that the final model will always be re-trained on the whole dataset!
```stan theme={null}
train_classification(ds, {
"target": "label_col",
"model": "CatboostClassifier",
"params": {
"boosting_type": "ordered"
},
"tune": {
"strategy": "grid",
"params": {
"learning_rate": [0.03, 0.1],
"depth": [4, 6, 10]
},
"validate": {
"n_splits": 5,
"metrics": ["f1_weighted", "accuracy"]
},
"scorer": "f1_weighted"
},
"validate": {
"n_splits": 1,
"test_size": 0.2
}
}) -> (ds.predicted, model)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
train_classification_gpu(ds: dataset, {
"param": value,
...
}) -> (*predicted: column, model: model_classification[ds])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Should contain the target column and the feature columns you wish to use in the model.
One or two columns containing the model's predictions. If two column names are provided, the second column
will contain the model's predicted probabilities.
Zip file containing the trained model and associated information.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Train a [Catboost classifier](https://catboost.ai/docs/).
I.e. gradient boosted decision trees with support for categorical variables and missing values.
Target variable (labels).
Name of the column that contains your target values (labels).
Name of the positive class.
In *binary* classification, usually the class you're most interested in, for example the label/class
corresponding to successful lead conversion in a lead score model, the class corresponding to a
customer who has churned in a churn prediction model, etc.
If provided, will automaticall measure the performance (accuracy, precision, recall) of the model on this
class, in addition to averages across all classes. If not provided, only summary metrics will be reported.
Maximum number of classes in the target variable.
If there are more classes than this, the least frequent classes will be grouped together into a single class
called "others". Reducing the number of classes in the target variable can help improve model performance,
especially when the number of classes is very large, some classes are very rare, or the dataset doesn't have
sufficient samples for all classes. Raising this significantly might lead to much longer training times.
Values must be in the following range:
```javascript theme={null}
2 ≤ max_classes ≤ 100
```
Importance of each feature in the model.
Whether and how to measure each feature's contribution to the model's predictions. The higher the value,
the more important the feature was in the model. Only relative values are meaningful, i.e. the importance
of a feature relative to other features in the model.
Also note that feature importance is usually meaningful only for models that fit the data well.
The default (`null`, `true` or `"native"`) uses the classifier's native feature importance measure, e.g.
[prediction-value-change](https://catboost.ai/en/docs/concepts/fstr#regular-feature-importance) in the case
of Catboost, [Gini importance](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier.feature_importances_)
in the case of scikit-learn's DecisionTreeClassifier, and the mean of absolute coefficients in the case of
logistic regression.
When set to `"permutation"`, uses [permutation importance](https://scikit-learn.org/stable/modules/permutation_importance.html),
i.e. measures the decrease in model score when a single feature's values are randomly shuffled. This is
considerably slower than native feature importance (the model needs to be evaluated an additional k\*n times,
where k is the number of features and n the number of repetitions to average over). On the positive side it is
model-agnostic and doesn't suffer from bias towards high cardinality features (like some tree-based feature
importances). On the negative side, it can be sensitive to strongly correlated features, as the unshuffled
correlated variable is still available to the model when shuffling the original variable.
When set to `false`, no feature importance will be calculated.
Values must be one of the following:
* `True`
* `False`
* `native`
* `permutation`
* `null`
Toggle encoding of feature columns.
When enabled, Graphext will auto-convert any column types to the numeric type before
fitting the model. How this conversion is done can be configured using the `feature_encoder`
option below.
If disabled, any model trained in this step will assume that input data
is already in an appropriate format (e.g. numerical and not containing any missing values).
Configures encoding of feature columns.
By default (`null`), Graphext chooses automatically how to convert any column types the model
may not understand natively to a numeric type.
A configuration object can be passed instead to overwrite specific parameter values with respect
to their default values.
Numeric encoder.
Configures encoding of numeric features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Further parameters passed to the `scaler` function.
Details depend no the particular scaler used.
Boolean encoder.
Configures encoding of boolean features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Ordinal encoder.
Configures encoding of categorical features that have a natural order.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Category encoder.
May contain either a single configuration for all categorical variables, or two different configurations
for low- and high-cardinality variables. For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for categories with fewer than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Multilabel encoder.
Configures encoding of multivalued categorical features (variable length lists of categories,
or the semantic type `list[category]` for short). May contain either a single configuration for
all multilabel variables, or two different configurations for low- and high-cardinality variables.
For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for mulitabel columns with fewer than `cardinality_threshold` unique categories/labels.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Datetime encoder.
Configures encoding of datetime (timestamp) features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
A list of numerical components to extract.
Will create one numeric column for each component.
Each item in array.
Values must be one of the following:
`day` `dayofweek` `dayofyear` `hour` `minute` `month` `quarter` `season` `second` `week` `weekday` `weekofyear` `year`
A list of cyclical time features to extract.
"Cycles" are numerical transformations of features that should be represented on a circle. E.g. months,
ranging from 1 to 12, should be arranged such that 12 and 1 are next to each other, rather than on
opposite ends of a linear scale. We represent such cyclical time features on a circle by creating two
columns for each original feature: the sin and cos of the numerical feature after appropriate scaling.
Each item in array.
Values must be one of the following:
* `day`
* `dayofweek`
* `dayofyear`
* `hour`
* `month`
Whether to include the epoch as new feature (seconds since 01/01/1970).
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Embedding/vector encoder.
Configures encoding of multivalued numerical features (variable length lists of numbers, i.e. vectors, or the semantic type `list[number]` for short).
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Text encoder.
Configures encoding of text (natural language) features. Currently only allows
[Tf-Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) embeddings to represent texts. If you wish
to use other embeddings, e.g. semantic, Word2Vec etc., transform your text column first using
another step, and then use that result instead of the original texts.
Texts are *excluded* by default from the overall encoding of the dataset. See parameter
`include_text_features` below to active it.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Parameters to be passed to the text encoder (Tf-Idf parameters only for now).
See [scikit-learn's documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)
for detailed parameters and their explanation.
How many output features to generate.
The resulting Tf-Idf vectors will be reduced to these many dimensions (columns) using scikit-learn's
[truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
This performs a kind of [latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
By default we will reduce to 200 components.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_components ≤ 1024
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Whether to include or ignore text columns during the processing of input data.
Enabling this will convert texts to their TfIdf representation. Each text will be
converted to an N-dimensional vector in which each component measures the relative
"over-representation" of a specific word (or n-gram) relative to its overall
frequency in the whole dataset. This is disabled by default because it will
often be better to convert texts explicitly using a previous step, such as
`embed_text` or `embed_text_with_model`.
CatBoost configuration parameters.
You can check the official documentation for more details about Catboost's parameters [here](https://catboost.ai/en/docs/references/training-parameters/).
The maximum depth of the tree.
Values must be in the following range:
```javascript theme={null}
2 ≤ depth ≤ 16
```
Number of iterations.
The maximum number of trees that can be built when solving machine learning problems. When using other parameters that limit the number of iterations, the final number of trees may be less than the number specified in this parameter.
Values must be in the following range:
```javascript theme={null}
1 ≤ iterations < inf
```
Maximum cardinality of variables to be one-hot encoded.
Use one-hot encoding for all categorical features with a number of different values less than or equal to this value. Other variables will be target-encoded. Note that one-hot encoding is faster than the alternatives, so decreasing this value makes it more likely slower methods will be used. See [CatBoost details](https://catboost.ai/docs/concepts/algorithm-main-stages_cat-to-numberic.html) for further information.
Values must be in the following range:
```javascript theme={null}
2 ≤ one_hot_max_size < inf
```
The maximum number of features that can be combined when transforming categorical variables.
Each resulting combination consists of one or more categorical features and can optionally contain binary features in the following form: “numeric feature > value”.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_ctr_complexity ≤ 4
```
Coefficient at the L2 regularization term of the cost function.
Values must be in the following range:
```javascript theme={null}
0.0 < l2_leaf_reg < inf
```
The number of splits for numerical features.
Values must be in the following range:
```javascript theme={null}
1 ≤ border_count ≤ 65535
```
The amount of randomness to use for scoring splits.
Use this parameter to avoid overfitting the model. The value multiplies the variance of a random variable (with
zero mean) that is added to the score used to select splits when a tree is grown.
Values must be in the following range:
```javascript theme={null}
0 < random_strength < inf
```
The method for processing missing values in the input dataset.
Possible values:
* “Forbidden”:
Missing values are not supported, their presence is interpreted as an error.
* “Min”:
Missing values are processed as the minimum value (less than all other values) for the feature.
It is guaranteed that a split that separates missing values from all other values is considered
when selecting trees.
* “Max”:
Missing values are processed as the maximum value (greater than all other values) for the feature.
It is guaranteed that a split that separates missing values from all other values is considered when
selecting trees.
Using the Min or Max value of this parameter guarantees that a split between missing values and other
values is considered when selecting a new split in the tree.
Values must be one of the following:
* `Forbidden`
* `Min`
* `Max`
Boosting type.
Boosting scheme. Possible values are
* Ordered: Usually provides better quality on small datasets, but it may be slower than the Plain scheme.
* Plain: The classic gradient boosting scheme.
Values must be one of the following:
* `Ordered`
* `Plain`
Random subspace method.
The percentage of features to use at each split selection, when features are selected over again at random. The value `null` is equivalent to 1.0 (all features). You can set this to values \< 1.0 when the dataset has many features (e.g. > 20) to speed up training.
Values must be in the following range:
```javascript theme={null}
0 < rsm ≤ 1.0
```
The random seed used for training.
Whether and how to limit memory usage.
Select the maximum Ram used using strings like "2GB" or "100mb" (non case\_sensitive).
Whether and how to assign weights to different predicted classes.
The options are:
* null: No class weighting
* Balanced: Inversely proportional to the number of samples/rows in each class
* SqrtBalanced: Using the square root of the "Balanced" option.
Values must be one of the following:
* `Balanced`
* `SqrtBalanced`
* `None`
* `None`
Configure model validation.
Allows evaluation of model performance via [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html)
using custom metrics. If not specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_splits < inf
```
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [k-fold cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#k-fold)
to split the dataset. E.g. if `n_splits` is 5, the dataset will be split into 5 equal-sized parts.
For five iterations four parts will then be used for training and the remaining part for testing.
If `test_size` is a number between 0 and 1, in contrast, validation is done using a
[shuffle-split](https://scikit-learn.org/stable/modules/cross_validation.html#shufflesplit)
approach. Here, instead of splitting the data into `n_splits` equal parts up front, in each iteration
we randomize the data and sample a proportion equal to `test_size` to use for evaluation and the remaining
rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
Whether to split the data by time.
Most recent data will be used for testing and previous data for training. Assumes data is passed
already sorted ascending by time.
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `roc_auc` `roc_auc_ovr` `roc_auc_ovo` `roc_auc_ovr_weighted` `roc_auc_ovo_weighted`
Configure hypertuning.
Configures the optimization of model hyper-parameters via cross-validated grid- or randomized search.
Which search strategy to use for optimization.
Grid search explores all possible combinations of parameters specified in `params`.
Randomized search, on the other hand, randomly samples `iterations` parameter combinations
from the distributions specified in `params`.
Values must be one of the following:
* `grid`
* `random`
How many randomly sampled parameter combinations to test in randomized search.
Values must be in the following range:
```javascript theme={null}
1 < iterations < inf
```
Configure model validation.
Allows evaluation of model performance via [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html)
using custom metrics. If not specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_splits < inf
```
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [k-fold cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#k-fold)
to split the dataset. E.g. if `n_splits` is 5, the dataset will be split into 5 equal-sized parts.
For five iterations four parts will then be used for training and the remaining part for testing.
If `test_size` is a number between 0 and 1, in contrast, validation is done using a
[shuffle-split](https://scikit-learn.org/stable/modules/cross_validation.html#shufflesplit)
approach. Here, instead of splitting the data into `n_splits` equal parts up front, in each iteration
we randomize the data and sample a proportion equal to `test_size` to use for evaluation and the remaining
rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
Whether to split the data by time.
Most recent data will be used for testing and previous data for training. Assumes data is passed
already sorted ascending by time.
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `roc_auc` `roc_auc_ovr` `roc_auc_ovo` `roc_auc_ovr_weighted` `roc_auc_ovo_weighted`
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `roc_auc` `roc_auc_ovr` `roc_auc_ovo` `roc_auc_ovr_weighted` `roc_auc_ovo_weighted`
The parameter values to explore.
Allows tuning of any and all of the parameters that can be set also as constants in the
"params" attribute.
Keys in this object should be strings identifying parameter names, and values should be
*lists* of values to explore for that parameter. E.g. `"depth": [3, 5, 7]`.
List of depths values to explore.
The maximum depth of the tree.
Values must be in the following range:
```javascript theme={null}
2 ≤ Item ≤ 16
```
List of iterations values to explore.
Number of iterations.
The maximum number of trees that can be built when solving machine learning problems. When using other parameters that limit the number of iterations, the final number of trees may be less than the number specified in this parameter.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item < inf
```
List of values configuring max cardinality for one-hot encoding.
Maximum cardinality of variables to be one-hot encoded.
Use one-hot encoding for all categorical features with a number of different values less than or equal to this value. Other variables will be target-encoded. Note that one-hot encoding is faster than the alternatives, so decreasing this value makes it more likely slower methods will be used. See [CatBoost details](https://catboost.ai/docs/concepts/algorithm-main-stages_cat-to-numberic.html) for further information.
Values must be in the following range:
```javascript theme={null}
2 ≤ Item < inf
```
List of values configuring variable combination complexity.
The maximum number of features that can be combined when transforming categorical variables.
Each resulting combination consists of one or more categorical features and can optionally contain binary features in the following form: “numeric feature > value”.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item ≤ 4
```
List of leaf regularization strengths.
Coefficient at the L2 regularization term of the cost function.
Values must be in the following range:
```javascript theme={null}
0.0 < Item < inf
```
List of border counts.
The number of splits for numerical features.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item ≤ 65535
```
List of random strengths.
The amount of randomness to use for scoring splits.
Use this parameter to avoid overfitting the model. The value multiplies the variance of a random variable (with
zero mean) that is added to the score used to select splits when a tree is grown.
Values must be in the following range:
```javascript theme={null}
0 < Item < inf
```
List of criterion values to explore.
Function to measure the quality of a split.
Supported criteria are "gini" for the Gini impurity and "entropy" for the information gain.
Values must be one of the following:
* `gini`
* `entropy`
List of splitter values to explore.
Strategy used to choose the split at each node.
Supported strategies are "best" to choose the best split and "random" to choose the best random split.
Values must be one of the following:
* `best`
* `random`
List of max\_depth values to explore.
Maximum depth of the tree.
The maximum depth of the tree. If null, then nodes are expanded until all leaves are pure
or until all leaves contain less than min\_samples\_split samples.
Values must be in the following range:
```javascript theme={null}
1 < Item < inf
```
List of min\_samples\_split values to explore.
Minimum number of samples required to split an internal node.
The minimum number of samples required to split an internal node. If int, then consider min\_samples\_split
as the minimum *count*. If float, then min\_samples\_split is a *fraction* and `ceil(min_samples_split * n_samples)`
are the minimum number of samples for each split.
List of min\_samples\_leaf values to explore.
Minimum number of samples required to be at a leaf node.
The minimum number of samples required to be at a leaf node. A split point at any depth will only be
considered if it leaves at least `min_samples_leaf` training samples in each of the left and right branches.
This may have the effect of smoothing the model, especially in regression.
If int, then consider `min_samples_leaf` as the minimum *count*. If float, then `min_samples_leaf` is
a *fraction* and `ceil(min_samples_leaf * n_samples)` are the minimum number of samples for each node.
List of max\_leaf\_nodes values to explore.
Grow a tree with `max_leaf_nodes` in best-first fashion.
Best nodes are defined as relative reduction in impurity. If null then unlimited number of leaf nodes.
List of max\_features values to explore.
Number of features to consider when looking for the best split.
The number of features to consider when looking for the best split:
* If int, then consider `max_features` features at each split.
* If float, then `max_features` is a *fraction* and `int(max_features * n_features)` features are considered at each split.
* If "auto", then `max_features=sqrt(n_features)`.
* If "sqrt", then `max_features=sqrt(n_features)`.
* If "log2", then `max_features=log2(n_features)`.
* If null, then `max_features=n_features`.
Note: the search for a split does not stop until at least one valid partition of the node samples is found,
even if it requires to effectively inspect more than `max_features` features.
List of ccp\_alpha values to explore.
Complexity parameter used for Minimal Cost-Complexity Pruning.
Minimal Cost-Complexity Pruning recursively finds the node with the "weakest link". The weakest link is
characterized by an effective alpha, where the nodes with the smallest effective alpha are pruned first.
As alpha increases, more of the tree is pruned, which increases the total impurity of its leaves.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ Item < inf
```
Seed for random number generator ensuring reproducibility.
Values must be in the following range:
```javascript theme={null}
0 ≤ seed < inf
```
Sort the data before training.
If the data is not already sorted by time, you can sort it here. This is useful when you want to split the data
by time, for example to train on older data and test on newer data (see the `time_split` parameter in validation
configurations). If the data is already sorted by time, you can ignore this parameter.
One or more column to sort by.
Each item in array.
Sort order.
Whether to sort in ascending or descending order. If the single value `true` is provided, or no value is specified,
all columns will be sorted in ascending order. If a single `false` is provided, all columns will be sorted in descending
order. If an array of booleans is provided, each column will be sorted according to the corresponding boolean value.
Each item in array.
# train_clustering
Source: https://docs.graphext.com/api-docs/analyse/train_and_predict/train_clustering
Train and store a machine learning model to be loaded at a later point for prediction.
Density-based clustering with ["HDBSCAN"](https://hdbscan.readthedocs.io/en/latest/how_hdbscan_works.html)
Generates a hierarchy of clusters, but then automatically selects the best *flat* clustering based on the stability
of clusters across a range of density thresholds. Roughly speaking, if a cluster's subclusters persists over a larger
range of the density parameter then the parent cluster itself, the subclusters will be selected, otherwise the parent.
The main parameter influencing cluster selection is `min_cluster_size`.
Can be used to predict the clusters of new data without changing the existing clustering.
## Usage
The following example shows how the step can be used in a recipe.
Train an HDBSCAN model with default parameters.
```stan theme={null}
train_clustering(ds) -> (ds.predicted, model)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
train_clustering(ds: dataset, {
"param": value,
...
}) -> (predicted: category, model: model_clustering[ds])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Should contain the target column and the feature columns you wish to use in the model.
Column containing results of the model.
Zip file containing the trained model and associated information.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Toggle encoding of feature columns.
When enabled, Graphext will auto-convert any column types to the numeric type before
fitting the model. How this conversion is done can be configured using the `feature_encoder`
option below.
If disabled, any model trained in this step will assume that input data
is already in an appropriate format (e.g. numerical and not containing any missing values).
Configures encoding of feature columns.
By default (`null`), Graphext chooses automatically how to convert any column types the model
may not understand natively to a numeric type.
A configuration object can be passed instead to overwrite specific parameter values with respect
to their default values.
Numeric encoder.
Configures encoding of numeric features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Further parameters passed to the `scaler` function.
Details depend no the particular scaler used.
Boolean encoder.
Configures encoding of boolean features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Ordinal encoder.
Configures encoding of categorical features that have a natural order.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Category encoder.
May contain either a single configuration for all categorical variables, or two different configurations
for low- and high-cardinality variables. For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for categories with fewer than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Multilabel encoder.
Configures encoding of multivalued categorical features (variable length lists of categories,
or the semantic type `list[category]` for short). May contain either a single configuration for
all multilabel variables, or two different configurations for low- and high-cardinality variables.
For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for mulitabel columns with fewer than `cardinality_threshold` unique categories/labels.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Datetime encoder.
Configures encoding of datetime (timestamp) features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
A list of numerical components to extract.
Will create one numeric column for each component.
Each item in array.
Values must be one of the following:
`day` `dayofweek` `dayofyear` `hour` `minute` `month` `quarter` `season` `second` `week` `weekday` `weekofyear` `year`
A list of cyclical time features to extract.
"Cycles" are numerical transformations of features that should be represented on a circle. E.g. months,
ranging from 1 to 12, should be arranged such that 12 and 1 are next to each other, rather than on
opposite ends of a linear scale. We represent such cyclical time features on a circle by creating two
columns for each original feature: the sin and cos of the numerical feature after appropriate scaling.
Each item in array.
Values must be one of the following:
* `day`
* `dayofweek`
* `dayofyear`
* `hour`
* `month`
Whether to include the epoch as new feature (seconds since 01/01/1970).
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Embedding/vector encoder.
Configures encoding of multivalued numerical features (variable length lists of numbers, i.e. vectors, or the semantic type `list[number]` for short).
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Text encoder.
Configures encoding of text (natural language) features. Currently only allows
[Tf-Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) embeddings to represent texts. If you wish
to use other embeddings, e.g. semantic, Word2Vec etc., transform your text column first using
another step, and then use that result instead of the original texts.
Texts are *excluded* by default from the overall encoding of the dataset. See parameter
`include_text_features` below to active it.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Parameters to be passed to the text encoder (Tf-Idf parameters only for now).
See [scikit-learn's documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)
for detailed parameters and their explanation.
How many output features to generate.
The resulting Tf-Idf vectors will be reduced to these many dimensions (columns) using scikit-learn's
[truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
This performs a kind of [latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
By default we will reduce to 200 components.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_components ≤ 1024
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Whether to include or ignore text columns during the processing of input data.
Enabling this will convert texts to their TfIdf representation. Each text will be
converted to an N-dimensional vector in which each component measures the relative
"over-representation" of a specific word (or n-gram) relative to its overall
frequency in the whole dataset. This is disabled by default because it will
often be better to convert texts explicitly using a previous step, such as
`embed_text` or `embed_text_with_model`.
Model parameters.
Also see [official HDBSCAN documentation](https://hdbscan.readthedocs.io/en/latest/parameter_selection.html) for details.
The minimum size of clusters.
Intuitively, the smallest size grouping you wish to consider a cluster. When selecting a flat clustering from the cluster
hierarchy, splits that contain fewer points than this will be considered points "falling out" of a cluster rather than a
cluster splitting into two new clusters.
Values must be in the following range:
```javascript theme={null}
1 ≤ min_cluster_size < inf
```
Determines how conservative the clustering is.
The larger the value, the more points will be declared as noise, and clusters will be restricted to progressively
more dense areas.
Values must be in the following range:
```javascript theme={null}
1 ≤ min_samples < inf
```
Distance threshold.
Clusters below this value will be merged. If default parameters result in areas with a large number of micro-clusters,
this parameter can help merging these clusters together. For example, set the value to 0.5 if you don't want to separate
clusters that are less than 0.5 units apart (the distance distribution depends on your specific data).
Values must be in the following range:
```javascript theme={null}
0.0 ≤ cluster_selection_epsilon < inf
```
Method used to select clusters from the cluster hierarchy.
The default, "excess of mass" (`eom`), can sometimes pick one or two large clusters and then a number
of small extra clusters. If you're interested in a more fine-grained clustering with a larger number of more homogeously
sized clusters, you may prefer selecting leaf clustering (`leaf`).
Values must be one of the following:
* `eof`
* `leaf`
Seed for random number generator ensuring reproducibility.
Values must be in the following range:
```javascript theme={null}
0 ≤ seed < inf
```
# train_dimensionality_reduction
Source: https://docs.graphext.com/api-docs/analyse/train_and_predict/train_dimensionality_reduction
Train and store a machine learning model to be loaded at a later point for prediction.
Dimensionality reduction with ["Uniform Manifold Approximation and Projection" (UMAP)](https://umap-learn.readthedocs.io/en/latest/)
Generates numeric embeddings (vectors) of the input data with reduced dimensionality, preserving
local and global similarities between data points. Can be used for visualisation, for example,
to arrange data in 2 dimensions according to their similarity, or to create nearest neighbour graphs/networks
(also see step `link_embeddings` in the latter case).
Can be used in supervised mode (providing a `target` column as parameter) or unsupervised (without target).
The output will always be a new column with the trained model's predictions on the training data,
as well as a saved and named model file that can be used in other projects for prediction of new data.
## Usage
The following examples show how the step can be used in a recipe.
Train an unsupervised UMAP model.
```stan theme={null}
train_embeddings(ds) -> (ds.predicted, model)
```
Train a supervised UMAP model.
```stan theme={null}
train_embeddings(ds, {"target": "reference"}) -> (ds.predicted, model)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
train_dimensionality_reduction(ds: dataset, {
"param": value,
...
}) -> (predicted: list[number], model: model_dimensionality_reduction[ds])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Should contain the target column and the feature columns you wish to use in the model.
Column containing results of the model.
Zip file containing the trained model and associated information.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Target variable.
Name of the column that contains your target values.
Toggle encoding of feature columns.
When enabled, Graphext will auto-convert any column types to the numeric type before
fitting the model. How this conversion is done can be configured using the `feature_encoder`
option below.
If disabled, any model trained in this step will assume that input data
is already in an appropriate format (e.g. numerical and not containing any missing values).
Configures encoding of feature columns.
By default (`null`), Graphext chooses automatically how to convert any column types the model
may not understand natively to a numeric type.
A configuration object can be passed instead to overwrite specific parameter values with respect
to their default values.
Numeric encoder.
Configures encoding of numeric features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Further parameters passed to the `scaler` function.
Details depend no the particular scaler used.
Boolean encoder.
Configures encoding of boolean features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Ordinal encoder.
Configures encoding of categorical features that have a natural order.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Category encoder.
May contain either a single configuration for all categorical variables, or two different configurations
for low- and high-cardinality variables. For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for categories with fewer than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Multilabel encoder.
Configures encoding of multivalued categorical features (variable length lists of categories,
or the semantic type `list[category]` for short). May contain either a single configuration for
all multilabel variables, or two different configurations for low- and high-cardinality variables.
For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for mulitabel columns with fewer than `cardinality_threshold` unique categories/labels.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Datetime encoder.
Configures encoding of datetime (timestamp) features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
A list of numerical components to extract.
Will create one numeric column for each component.
Each item in array.
Values must be one of the following:
`day` `dayofweek` `dayofyear` `hour` `minute` `month` `quarter` `season` `second` `week` `weekday` `weekofyear` `year`
A list of cyclical time features to extract.
"Cycles" are numerical transformations of features that should be represented on a circle. E.g. months,
ranging from 1 to 12, should be arranged such that 12 and 1 are next to each other, rather than on
opposite ends of a linear scale. We represent such cyclical time features on a circle by creating two
columns for each original feature: the sin and cos of the numerical feature after appropriate scaling.
Each item in array.
Values must be one of the following:
* `day`
* `dayofweek`
* `dayofyear`
* `hour`
* `month`
Whether to include the epoch as new feature (seconds since 01/01/1970).
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Embedding/vector encoder.
Configures encoding of multivalued numerical features (variable length lists of numbers, i.e. vectors, or the semantic type `list[number]` for short).
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Text encoder.
Configures encoding of text (natural language) features. Currently only allows
[Tf-Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) embeddings to represent texts. If you wish
to use other embeddings, e.g. semantic, Word2Vec etc., transform your text column first using
another step, and then use that result instead of the original texts.
Texts are *excluded* by default from the overall encoding of the dataset. See parameter
`include_text_features` below to active it.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Parameters to be passed to the text encoder (Tf-Idf parameters only for now).
See [scikit-learn's documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)
for detailed parameters and their explanation.
How many output features to generate.
The resulting Tf-Idf vectors will be reduced to these many dimensions (columns) using scikit-learn's
[truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
This performs a kind of [latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
By default we will reduce to 200 components.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_components ≤ 1024
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Whether to include or ignore text columns during the processing of input data.
Enabling this will convert texts to their TfIdf representation. Each text will be
converted to an N-dimensional vector in which each component measures the relative
"over-representation" of a specific word (or n-gram) relative to its overall
frequency in the whole dataset. This is disabled by default because it will
often be better to convert texts explicitly using a previous step, such as
`embed_text` or `embed_text_with_model`.
Model parameters.
See [official UMAP documentation](https://umap-learn.readthedocs.io/en/latest/parameters.html) for details.
Number of neighbors.
This determines the number of neighboring points used in local approximations of manifold structure.
Larger values will result in more global structure being preserved at the loss of detailed local
structure. In general this parameter should often be in the range 5 to 50, with a choice of 10 to 15
being a sensible default.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_neighbors < inf
```
Minimum distance between reduced data points.
Controls how tightly UMAP is allowed to pack points together in the reduced space. Smaller values will lead to points more tightly
packed together (potentially useful if result is used to cluster the points). Larger values will distribute points with more space
between them (which may be desirable for visualization, or to focus more on the global structure of the date).
For further details see [here](https://umap-learn.readthedocs.io/en/latest/parameters.html#min-dist).
Values must be in the following range:
```javascript theme={null}
0 ≤ min_dist < inf
```
Number of n\_components.
Allows the user to determine the dimensionality of the reduced dimension space we will be embedding the
data into.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_components < inf
```
Metric to use for measuring similarity between data points.
Values must be one of the following:
`euclidean` `manhattan` `chebyshev` `minkowski` `canberra` `braycurtis` `haversine` `mahalanobis` `wminkowski` `seuclidean` `cosine` `correlation` `hamming` `jaccard` `dice` `russellrao` `kulsinski` `rogerstanimoto` `sokalmichener` `sokalsneath` `yule`
Number of training iterations used in optimizing the embedding.
Larger values result in more accurate embeddings. If `null` is specified a value will be selected based on the size of the input dataset
(200 for large datasets, 500 for small).
How to initialize the low dimensional embedding.
When "spectral", uses a (relatively expensive) spectral embedding. "pca" uses the first `n_components`
from a principal component analysis. "tswspectral" is a cheaper alternative to "spectral". When "random",
assigns initial embedding positions at random. This uses the least amount of memory and time but may make UMAP
slower to converge on the optimal embedding.
Values must be one of the following:
* `spectral`
* `pca`
* `tswspectral`
* `random`
Avoid excessive memory use.
For some datasets nearest neighbor computations can consume a lot of memory. If you find the step is failing due to memory constraints,
consider setting this option to `true`. This approach is more computationally expensive, but avoids excessive memory use.
Weighting factor between features and target.
A value of 0.0 weights entirely on data, and a value of 1.0 weights entirely on target. The default of 0.5 balances
the weighting equally between data and target.
Seed for random number generator ensuring reproducibility.
Values must be in the following range:
```javascript theme={null}
0 ≤ seed < inf
```
# train_regression
Source: https://docs.graphext.com/api-docs/analyse/train_and_predict/train_regression
Train and store a regression model to be loaded at a later point for prediction.
Note that the configuration parameters depend on the specific model trained. In the parameters
section below, each allowed model has its own section.
The output will always be a new column with the trained model's predictions on the training data,
as well as a saved and named model file that can be used in other projects for prediction of new data.
A detailed guide on how to configure this step for model tuning and performance evaluation can be found
[here](/guides/model_train_eval/).
## Usage
The following examples show how the step can be used in a recipe.
Train a regression model with default parameters. By default, a Catboost model will be trained, but this can be changed to any of the supported models by specifying the `model` parameter (see below for details):
```stan theme={null}
train_regression(ds, {"target": "class"}) -> (ds.predicted, "my-regr")
```
To be more explicit about which model parameters to use during training, which parameters to optimize (tune) automatically, and how to evaluate the model's performance, the following example shows a complete configuration. It will explicitly select the CatboostRegressor as the model, set `boosting_type` to "ordered", and select the best combination of `learning_rate` and `depth` from the values specified in the `tune: params` configuration. To find the best parameters, it will perform 5-fold cross-validation on each combination, and will use the scorer `f1_weighted` to measure the performance. `accuracy` will also be measured, but only for the purpose of reporting. The best parameter combination will than be evaluated on a single split of the dataset (with 20% of rows used for testing and 80% for training), with metrics selected automatically. Note that the final model will always be re-trained on the whole dataset!
```stan theme={null}
train_regression(ds, {
"target": "label_col",
"model": "CatboostRegressor",
"params": {
"boosting_type": "ordered"
},
"tune": {
"strategy": "grid",
"params": {
"learning_rate": [0.03, 0.1],
"depth": [4, 6, 10]
},
"validate": {
"n_splits": 5,
"metrics": ["f1_weighted", "accuracy"]
},
"scorer": "f1_weighted"
},
"validate": {
"n_splits": 1,
"test_size": 0.2
}
}) -> (ds.predicted, "my-regr")
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
train_regression(ds: dataset, {
"param": value,
...
}) -> (predicted: number, model: model_regression[ds])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Should contain the target column and the feature columns you wish to use in the model.
Column containing results of the model.
Zip file containing the trained model and associated information.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Train a [Catboost regressor](https://catboost.ai/docs/).
I.e. gradient boosted decision trees with support for categorical variables and missing values.
Target variable (labels).
Name of the column that contains your target values (labels).
Importance of each feature in the model.
Whether and how to measure each feature's contribution to the model's predictions. The higher the value,
the more important the feature was in the model. Only relative values are meaningful, i.e. the importance
of a feature relative to other features in the model.
Also note that feature importance is usually meaningful only for models that fit the data well.
The default (`null`, `true` or `"native"`) uses the classifier's native feature importance measure, e.g.
[prediction-value-change](https://catboost.ai/en/docs/concepts/fstr#regular-feature-importance) in the case
of Catboost, [Gini importance](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier.feature_importances_)
in the case of scikit-learn's DecisionTreeClassifier, and the mean of absolute coefficients in the case of
logistic regression.
When set to `"permutation"`, uses [permutation importance](https://scikit-learn.org/stable/modules/permutation_importance.html),
i.e. measures the decrease in model score when a single feature's values are randomly shuffled. This is
considerably slower than native feature importance (the model needs to be evaluated an additional k\*n times,
where k is the number of features and n the number of repetitions to average over). On the positive side it is
model-agnostic and doesn't suffer from bias towards high cardinality features (like some tree-based feature
importances). On the negative side, it can be sensitive to strongly correlated features, as the unshuffled
correlated variable is still available to the model when shuffling the original variable.
When set to `false`, no feature importance will be calculated.
Values must be one of the following:
* `True`
* `False`
* `native`
* `permutation`
* `null`
Toggle encoding of feature columns.
When enabled, Graphext will auto-convert any column types to the numeric type before
fitting the model. How this conversion is done can be configured using the `feature_encoder`
option below.
If disabled, any model trained in this step will assume that input data
is already in an appropriate format (e.g. numerical and not containing any missing values).
Configures encoding of feature columns.
By default (`null`), Graphext chooses automatically how to convert any column types the model
may not understand natively to a numeric type.
A configuration object can be passed instead to overwrite specific parameter values with respect
to their default values.
Numeric encoder.
Configures encoding of numeric features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Further parameters passed to the `scaler` function.
Details depend no the particular scaler used.
Boolean encoder.
Configures encoding of boolean features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Ordinal encoder.
Configures encoding of categorical features that have a natural order.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Category encoder.
May contain either a single configuration for all categorical variables, or two different configurations
for low- and high-cardinality variables. For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for categories with fewer than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Multilabel encoder.
Configures encoding of multivalued categorical features (variable length lists of categories,
or the semantic type `list[category]` for short). May contain either a single configuration for
all multilabel variables, or two different configurations for low- and high-cardinality variables.
For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for mulitabel columns with fewer than `cardinality_threshold` unique categories/labels.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Datetime encoder.
Configures encoding of datetime (timestamp) features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
A list of numerical components to extract.
Will create one numeric column for each component.
Each item in array.
Values must be one of the following:
`day` `dayofweek` `dayofyear` `hour` `minute` `month` `quarter` `season` `second` `week` `weekday` `weekofyear` `year`
A list of cyclical time features to extract.
"Cycles" are numerical transformations of features that should be represented on a circle. E.g. months,
ranging from 1 to 12, should be arranged such that 12 and 1 are next to each other, rather than on
opposite ends of a linear scale. We represent such cyclical time features on a circle by creating two
columns for each original feature: the sin and cos of the numerical feature after appropriate scaling.
Each item in array.
Values must be one of the following:
* `day`
* `dayofweek`
* `dayofyear`
* `hour`
* `month`
Whether to include the epoch as new feature (seconds since 01/01/1970).
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Embedding/vector encoder.
Configures encoding of multivalued numerical features (variable length lists of numbers, i.e. vectors, or the semantic type `list[number]` for short).
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Text encoder.
Configures encoding of text (natural language) features. Currently only allows
[Tf-Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) embeddings to represent texts. If you wish
to use other embeddings, e.g. semantic, Word2Vec etc., transform your text column first using
another step, and then use that result instead of the original texts.
Texts are *excluded* by default from the overall encoding of the dataset. See parameter
`include_text_features` below to active it.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Parameters to be passed to the text encoder (Tf-Idf parameters only for now).
See [scikit-learn's documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)
for detailed parameters and their explanation.
How many output features to generate.
The resulting Tf-Idf vectors will be reduced to these many dimensions (columns) using scikit-learn's
[truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
This performs a kind of [latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
By default we will reduce to 200 components.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_components ≤ 1024
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Whether to include or ignore text columns during the processing of input data.
Enabling this will convert texts to their TfIdf representation. Each text will be
converted to an N-dimensional vector in which each component measures the relative
"over-representation" of a specific word (or n-gram) relative to its overall
frequency in the whole dataset. This is disabled by default because it will
often be better to convert texts explicitly using a previous step, such as
`embed_text` or `embed_text_with_model`.
CatBoost configuration parameters.
You can check the official documentation for more details about Catboost's parameters [here](https://catboost.ai/en/docs/references/training-parameters/).
The maximum depth of the tree.
Values must be in the following range:
```javascript theme={null}
2 ≤ depth ≤ 16
```
Number of iterations.
The maximum number of trees that can be built when solving machine learning problems. When using other parameters that limit the number of iterations, the final number of trees may be less than the number specified in this parameter.
Values must be in the following range:
```javascript theme={null}
1 ≤ iterations < inf
```
Maximum cardinality of variables to be one-hot encoded.
Use one-hot encoding for all categorical features with a number of different values less than or equal to this value. Other variables will be target-encoded. Note that one-hot encoding is faster than the alternatives, so decreasing this value makes it more likely slower methods will be used. See [CatBoost details](https://catboost.ai/docs/concepts/algorithm-main-stages_cat-to-numberic.html) for further information.
Values must be in the following range:
```javascript theme={null}
2 ≤ one_hot_max_size < inf
```
The maximum number of features that can be combined when transforming categorical variables.
Each resulting combination consists of one or more categorical features and can optionally contain binary features in the following form: “numeric feature > value”.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_ctr_complexity ≤ 4
```
Coefficient at the L2 regularization term of the cost function.
Values must be in the following range:
```javascript theme={null}
0.0 < l2_leaf_reg < inf
```
The number of splits for numerical features.
Values must be in the following range:
```javascript theme={null}
1 ≤ border_count ≤ 65535
```
The amount of randomness to use for scoring splits.
Use this parameter to avoid overfitting the model. The value multiplies the variance of a random variable (with
zero mean) that is added to the score used to select splits when a tree is grown.
Values must be in the following range:
```javascript theme={null}
0 < random_strength < inf
```
The method for processing missing values in the input dataset.
Possible values:
* “Forbidden”:
Missing values are not supported, their presence is interpreted as an error.
* “Min”:
Missing values are processed as the minimum value (less than all other values) for the feature.
It is guaranteed that a split that separates missing values from all other values is considered
when selecting trees.
* “Max”:
Missing values are processed as the maximum value (greater than all other values) for the feature.
It is guaranteed that a split that separates missing values from all other values is considered when
selecting trees.
Using the Min or Max value of this parameter guarantees that a split between missing values and other
values is considered when selecting a new split in the tree.
Values must be one of the following:
* `Forbidden`
* `Min`
* `Max`
Boosting type.
Boosting scheme. Possible values are
* Ordered: Usually provides better quality on small datasets, but it may be slower than the Plain scheme.
* Plain: The classic gradient boosting scheme.
Values must be one of the following:
* `Ordered`
* `Plain`
Random subspace method.
The percentage of features to use at each split selection, when features are selected over again at random. The value `null` is equivalent to 1.0 (all features). You can set this to values \< 1.0 when the dataset has many features (e.g. > 20) to speed up training.
Values must be in the following range:
```javascript theme={null}
0 < rsm ≤ 1.0
```
The random seed used for training.
Whether and how to limit memory usage.
Select the maximum Ram used using strings like "2GB" or "100mb" (non case\_sensitive).
Whether and how to assign weights to different predicted classes.
The options are:
* null: No class weighting
* Balanced: Inversely proportional to the number of samples/rows in each class
* SqrtBalanced: Using the square root of the "Balanced" option.
Values must be one of the following:
* `Balanced`
* `SqrtBalanced`
* `None`
* `None`
Configure model validation.
Allows evaluation of model performance via [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html)
using custom metrics. If not specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_splits < inf
```
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [k-fold cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#k-fold)
to split the dataset. E.g. if `n_splits` is 5, the dataset will be split into 5 equal-sized parts.
For five iterations four parts will then be used for training and the remaining part for testing.
If `test_size` is a number between 0 and 1, in contrast, validation is done using a
[shuffle-split](https://scikit-learn.org/stable/modules/cross_validation.html#shufflesplit)
approach. Here, instead of splitting the data into `n_splits` equal parts up front, in each iteration
we randomize the data and sample a proportion equal to `test_size` to use for evaluation and the remaining
rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
Whether to split the data by time.
Most recent data will be used for testing and previous data for training. Assumes data is passed
already sorted ascending by time.
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
Each item in array.
Values must be one of the following:
`explained_variance` `neg_mean_absolute_error` `neg_median_absolute_error` `neg_mean_squared_error` `neg_root_mean_squared_error` `r2`
Configure hypertuning.
Configures the optimization of model hyper-parameters via cross-validated grid- or randomized search.
Which search strategy to use for optimization.
Grid search explores all possible combinations of parameters specified in `params`.
Randomized search, on the other hand, randomly samples `iterations` parameter combinations
from the distributions specified in `params`.
Values must be one of the following:
* `grid`
* `random`
How many randomly sampled parameter combinations to test in randomized search.
Values must be in the following range:
```javascript theme={null}
1 < iterations < inf
```
Configure model validation.
Allows evaluation of model performance via [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html)
using custom metrics. If not specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_splits < inf
```
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [k-fold cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#k-fold)
to split the dataset. E.g. if `n_splits` is 5, the dataset will be split into 5 equal-sized parts.
For five iterations four parts will then be used for training and the remaining part for testing.
If `test_size` is a number between 0 and 1, in contrast, validation is done using a
[shuffle-split](https://scikit-learn.org/stable/modules/cross_validation.html#shufflesplit)
approach. Here, instead of splitting the data into `n_splits` equal parts up front, in each iteration
we randomize the data and sample a proportion equal to `test_size` to use for evaluation and the remaining
rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
Whether to split the data by time.
Most recent data will be used for testing and previous data for training. Assumes data is passed
already sorted ascending by time.
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
Each item in array.
Values must be one of the following:
`explained_variance` `neg_mean_absolute_error` `neg_median_absolute_error` `neg_mean_squared_error` `neg_root_mean_squared_error` `r2`
Metric used to select best model.
Values must be one of the following:
`explained_variance` `neg_mean_absolute_error` `neg_median_absolute_error` `neg_mean_squared_error` `neg_root_mean_squared_error` `r2`
The parameter values to explore.
Allows tuning of any and all of the parameters that can be set also as constants in the
"params" attribute.
Keys in this object should be strings identifying parameter names, and values should be
*lists* of values to explore for that parameter. E.g. `"depth": [3, 5, 7]`.
List of depths values to explore.
The maximum depth of the tree.
Values must be in the following range:
```javascript theme={null}
2 ≤ Item ≤ 16
```
List of iterations values to explore.
Number of iterations.
The maximum number of trees that can be built when solving machine learning problems. When using other parameters that limit the number of iterations, the final number of trees may be less than the number specified in this parameter.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item < inf
```
List of values configuring max cardinality for one-hot encoding.
Maximum cardinality of variables to be one-hot encoded.
Use one-hot encoding for all categorical features with a number of different values less than or equal to this value. Other variables will be target-encoded. Note that one-hot encoding is faster than the alternatives, so decreasing this value makes it more likely slower methods will be used. See [CatBoost details](https://catboost.ai/docs/concepts/algorithm-main-stages_cat-to-numberic.html) for further information.
Values must be in the following range:
```javascript theme={null}
2 ≤ Item < inf
```
List of values configuring variable combination complexity.
The maximum number of features that can be combined when transforming categorical variables.
Each resulting combination consists of one or more categorical features and can optionally contain binary features in the following form: “numeric feature > value”.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item ≤ 4
```
List of leaf regularization strengths.
Coefficient at the L2 regularization term of the cost function.
Values must be in the following range:
```javascript theme={null}
0.0 < Item < inf
```
List of border counts.
The number of splits for numerical features.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item ≤ 65535
```
List of random strengths.
The amount of randomness to use for scoring splits.
Use this parameter to avoid overfitting the model. The value multiplies the variance of a random variable (with
zero mean) that is added to the score used to select splits when a tree is grown.
Values must be in the following range:
```javascript theme={null}
0 < Item < inf
```
List of criterion values to explore.
Function to measure the quality of a split.
Supported criteria are "gini" for the Gini impurity and "entropy" for the information gain.
Values must be one of the following:
* `gini`
* `entropy`
List of splitter values to explore.
Strategy used to choose the split at each node.
Supported strategies are "best" to choose the best split and "random" to choose the best random split.
Values must be one of the following:
* `best`
* `random`
List of max\_depth values to explore.
Maximum depth of the tree.
The maximum depth of the tree. If null, then nodes are expanded until all leaves are pure
or until all leaves contain less than min\_samples\_split samples.
Values must be in the following range:
```javascript theme={null}
1 < Item < inf
```
List of min\_samples\_split values to explore.
Minimum number of samples required to split an internal node.
The minimum number of samples required to split an internal node. If int, then consider min\_samples\_split
as the minimum *count*. If float, then min\_samples\_split is a *fraction* and `ceil(min_samples_split * n_samples)`
are the minimum number of samples for each split.
List of min\_samples\_leaf values to explore.
Minimum number of samples required to be at a leaf node.
The minimum number of samples required to be at a leaf node. A split point at any depth will only be
considered if it leaves at least `min_samples_leaf` training samples in each of the left and right branches.
This may have the effect of smoothing the model, especially in regression.
If int, then consider `min_samples_leaf` as the minimum *count*. If float, then `min_samples_leaf` is
a *fraction* and `ceil(min_samples_leaf * n_samples)` are the minimum number of samples for each node.
List of max\_leaf\_nodes values to explore.
Grow a tree with `max_leaf_nodes` in best-first fashion.
Best nodes are defined as relative reduction in impurity. If null then unlimited number of leaf nodes.
List of max\_features values to explore.
Number of features to consider when looking for the best split.
The number of features to consider when looking for the best split:
* If int, then consider `max_features` features at each split.
* If float, then `max_features` is a *fraction* and `int(max_features * n_features)` features are considered at each split.
* If "auto", then `max_features=sqrt(n_features)`.
* If "sqrt", then `max_features=sqrt(n_features)`.
* If "log2", then `max_features=log2(n_features)`.
* If null, then `max_features=n_features`.
Note: the search for a split does not stop until at least one valid partition of the node samples is found,
even if it requires to effectively inspect more than `max_features` features.
List of ccp\_alpha values to explore.
Complexity parameter used for Minimal Cost-Complexity Pruning.
Minimal Cost-Complexity Pruning recursively finds the node with the "weakest link". The weakest link is
characterized by an effective alpha, where the nodes with the smallest effective alpha are pruned first.
As alpha increases, more of the tree is pruned, which increases the total impurity of its leaves.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ Item < inf
```
Seed for random number generator ensuring reproducibility.
Values must be in the following range:
```javascript theme={null}
0 ≤ seed < inf
```
Trains a linear regression.
The specific kind of linear regression trained here is an "elastic net", which allows for a blend
of ridge and lasso regularization to prevent overfitting. The mix as well as the strength of this
regularization is automatically tuned using 5-fold cross-validation. See
[sklearn's ElasticNetCV](https://scikit-learn.org/stable/modules/linear_model.html#elastic-net)
for further details.
Target variable (labels).
Name of the column that contains your target values (labels).
Toggle encoding of feature columns.
When enabled, Graphext will auto-convert any column types to the numeric type before
fitting the model. How this conversion is done can be configured using the `feature_encoder`
option below.
If disabled, any model trained in this step will assume that input data
is already in an appropriate format (e.g. numerical and not containing any missing values).
Configures encoding of feature columns.
By default (`null`), Graphext chooses automatically how to convert any column types the model
may not understand natively to a numeric type.
A configuration object can be passed instead to overwrite specific parameter values with respect
to their default values.
Numeric encoder.
Configures encoding of numeric features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Further parameters passed to the `scaler` function.
Details depend no the particular scaler used.
Boolean encoder.
Configures encoding of boolean features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Ordinal encoder.
Configures encoding of categorical features that have a natural order.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Category encoder.
May contain either a single configuration for all categorical variables, or two different configurations
for low- and high-cardinality variables. For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for categories with fewer than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Multilabel encoder.
Configures encoding of multivalued categorical features (variable length lists of categories,
or the semantic type `list[category]` for short). May contain either a single configuration for
all multilabel variables, or two different configurations for low- and high-cardinality variables.
For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for mulitabel columns with fewer than `cardinality_threshold` unique categories/labels.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Datetime encoder.
Configures encoding of datetime (timestamp) features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
A list of numerical components to extract.
Will create one numeric column for each component.
Each item in array.
Values must be one of the following:
`day` `dayofweek` `dayofyear` `hour` `minute` `month` `quarter` `season` `second` `week` `weekday` `weekofyear` `year`
A list of cyclical time features to extract.
"Cycles" are numerical transformations of features that should be represented on a circle. E.g. months,
ranging from 1 to 12, should be arranged such that 12 and 1 are next to each other, rather than on
opposite ends of a linear scale. We represent such cyclical time features on a circle by creating two
columns for each original feature: the sin and cos of the numerical feature after appropriate scaling.
Each item in array.
Values must be one of the following:
* `day`
* `dayofweek`
* `dayofyear`
* `hour`
* `month`
Whether to include the epoch as new feature (seconds since 01/01/1970).
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Embedding/vector encoder.
Configures encoding of multivalued numerical features (variable length lists of numbers, i.e. vectors, or the semantic type `list[number]` for short).
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Text encoder.
Configures encoding of text (natural language) features. Currently only allows
[Tf-Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) embeddings to represent texts. If you wish
to use other embeddings, e.g. semantic, Word2Vec etc., transform your text column first using
another step, and then use that result instead of the original texts.
Texts are *excluded* by default from the overall encoding of the dataset. See parameter
`include_text_features` below to active it.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Parameters to be passed to the text encoder (Tf-Idf parameters only for now).
See [scikit-learn's documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)
for detailed parameters and their explanation.
How many output features to generate.
The resulting Tf-Idf vectors will be reduced to these many dimensions (columns) using scikit-learn's
[truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
This performs a kind of [latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
By default we will reduce to 200 components.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_components ≤ 1024
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Whether to include or ignore text columns during the processing of input data.
Enabling this will convert texts to their TfIdf representation. Each text will be
converted to an N-dimensional vector in which each component measures the relative
"over-representation" of a specific word (or n-gram) relative to its overall
frequency in the whole dataset. This is disabled by default because it will
often be better to convert texts explicitly using a previous step, such as
`embed_text` or `embed_text_with_model`.
Model parameters.
Constant parameters to configure before training.
Relative weight of l1 norm penalty vs l2 norm penalty.
An l1-ratio of 0 means l2 penalty only (euclidean norm), resulting in a ridge regression penalizing large
coefficients proportional to their sum of squares. An l1-ratio of 1.0 means l1 penalty only (taxicab/manhattan norm),
i.e. proportional to the sum of absolute coefficient values. This has the tendency to prefer solutions with fewer
non-zero coefficients, effectively reducing the number of features used in the optimized model.
Each item in array.
Values must be in the following range:
```javascript theme={null}
0 < Item ≤ 1
```
Number of alphas (regularization strengths) to test for each l1\_ratio.
Values must be in the following range:
```javascript theme={null}
10 ≤ n_alphas < inf
```
Feature normalization.
Whether to normalize features before regression by subtracting the mean and dividing by the l2-norm. Note that by default feature will automatically pre-processed, so you may want to enables this only after disabling feature encoding first.
Maximum number of iterations of the optimization algorithm.
Try increasing this if you suspect the algorithm doesn't reach the peformance you'd expect.
Values must be in the following range:
```javascript theme={null}
100 ≤ max_iter < inf
```
Configure model validation.
Allows evaluation of model performance via [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html)
using custom metrics. If not specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_splits < inf
```
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [k-fold cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#k-fold)
to split the dataset. E.g. if `n_splits` is 5, the dataset will be split into 5 equal-sized parts.
For five iterations four parts will then be used for training and the remaining part for testing.
If `test_size` is a number between 0 and 1, in contrast, validation is done using a
[shuffle-split](https://scikit-learn.org/stable/modules/cross_validation.html#shufflesplit)
approach. Here, instead of splitting the data into `n_splits` equal parts up front, in each iteration
we randomize the data and sample a proportion equal to `test_size` to use for evaluation and the remaining
rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
Whether to split the data by time.
Most recent data will be used for testing and previous data for training. Assumes data is passed
already sorted ascending by time.
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
Each item in array.
Values must be one of the following:
`explained_variance` `neg_mean_absolute_error` `neg_median_absolute_error` `neg_mean_squared_error` `neg_root_mean_squared_error` `r2`
Seed for random number generator ensuring reproducibility.
Values must be in the following range:
```javascript theme={null}
0 ≤ seed < inf
```
Trains a decision tree regression.
A [decision tree](https://en.wikipedia.org/wiki/Decision_tree_learning) is a non-parametric,
supervised method for predicting a target variable by learning simple decision rules inferred
from the data. It can be seen as a piecewise constant approximation, applying simple if-else decision
rules to the data. The particular model used here is scikit-learn's
[DecisionTreeRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html).
Target variable (labels).
Name of the column that contains your target values (labels).
Importance of each feature in the model.
Whether and how to measure each feature's contribution to the model's predictions. The higher the value,
the more important the feature was in the model. Only relative values are meaningful, i.e. the importance
of a feature relative to other features in the model.
Also note that feature importance is usually meaningful only for models that fit the data well.
The default (`null`, `true` or `"native"`) uses the classifier's native feature importance measure, e.g.
[prediction-value-change](https://catboost.ai/en/docs/concepts/fstr#regular-feature-importance) in the case
of Catboost, [Gini importance](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier.feature_importances_)
in the case of scikit-learn's DecisionTreeClassifier, and the mean of absolute coefficients in the case of
logistic regression.
When set to `"permutation"`, uses [permutation importance](https://scikit-learn.org/stable/modules/permutation_importance.html),
i.e. measures the decrease in model score when a single feature's values are randomly shuffled. This is
considerably slower than native feature importance (the model needs to be evaluated an additional k\*n times,
where k is the number of features and n the number of repetitions to average over). On the positive side it is
model-agnostic and doesn't suffer from bias towards high cardinality features (like some tree-based feature
importances). On the negative side, it can be sensitive to strongly correlated features, as the unshuffled
correlated variable is still available to the model when shuffling the original variable.
When set to `false`, no feature importance will be calculated.
Values must be one of the following:
* `True`
* `False`
* `native`
* `permutation`
* `null`
Toggle encoding of feature columns.
When enabled, Graphext will auto-convert any column types to the numeric type before
fitting the model. How this conversion is done can be configured using the `feature_encoder`
option below.
If disabled, any model trained in this step will assume that input data
is already in an appropriate format (e.g. numerical and not containing any missing values).
Configures encoding of feature columns.
By default (`null`), Graphext chooses automatically how to convert any column types the model
may not understand natively to a numeric type.
A configuration object can be passed instead to overwrite specific parameter values with respect
to their default values.
Numeric encoder.
Configures encoding of numeric features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Further parameters passed to the `scaler` function.
Details depend no the particular scaler used.
Boolean encoder.
Configures encoding of boolean features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Ordinal encoder.
Configures encoding of categorical features that have a natural order.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Category encoder.
May contain either a single configuration for all categorical variables, or two different configurations
for low- and high-cardinality variables. For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for categories with fewer than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Multilabel encoder.
Configures encoding of multivalued categorical features (variable length lists of categories,
or the semantic type `list[category]` for short). May contain either a single configuration for
all multilabel variables, or two different configurations for low- and high-cardinality variables.
For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for mulitabel columns with fewer than `cardinality_threshold` unique categories/labels.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Datetime encoder.
Configures encoding of datetime (timestamp) features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
A list of numerical components to extract.
Will create one numeric column for each component.
Each item in array.
Values must be one of the following:
`day` `dayofweek` `dayofyear` `hour` `minute` `month` `quarter` `season` `second` `week` `weekday` `weekofyear` `year`
A list of cyclical time features to extract.
"Cycles" are numerical transformations of features that should be represented on a circle. E.g. months,
ranging from 1 to 12, should be arranged such that 12 and 1 are next to each other, rather than on
opposite ends of a linear scale. We represent such cyclical time features on a circle by creating two
columns for each original feature: the sin and cos of the numerical feature after appropriate scaling.
Each item in array.
Values must be one of the following:
* `day`
* `dayofweek`
* `dayofyear`
* `hour`
* `month`
Whether to include the epoch as new feature (seconds since 01/01/1970).
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Embedding/vector encoder.
Configures encoding of multivalued numerical features (variable length lists of numbers, i.e. vectors, or the semantic type `list[number]` for short).
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Text encoder.
Configures encoding of text (natural language) features. Currently only allows
[Tf-Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) embeddings to represent texts. If you wish
to use other embeddings, e.g. semantic, Word2Vec etc., transform your text column first using
another step, and then use that result instead of the original texts.
Texts are *excluded* by default from the overall encoding of the dataset. See parameter
`include_text_features` below to active it.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Parameters to be passed to the text encoder (Tf-Idf parameters only for now).
See [scikit-learn's documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)
for detailed parameters and their explanation.
How many output features to generate.
The resulting Tf-Idf vectors will be reduced to these many dimensions (columns) using scikit-learn's
[truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
This performs a kind of [latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
By default we will reduce to 200 components.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_components ≤ 1024
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Whether to include or ignore text columns during the processing of input data.
Enabling this will convert texts to their TfIdf representation. Each text will be
converted to an N-dimensional vector in which each component measures the relative
"over-representation" of a specific word (or n-gram) relative to its overall
frequency in the whole dataset. This is disabled by default because it will
often be better to convert texts explicitly using a previous step, such as
`embed_text` or `embed_text_with_model`.
Decision tree configuration parameters.
These parameters are specific to the decision tree algorithm. They are used to define the tree structure
and the stopping criteria. The default values are the ones used by scikit-learn.
For more information, see the [scikit-learn documentation](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier).
Function to measure the quality of a split.
Supported criteria are "gini" for the Gini impurity and "entropy" for the information gain.
Values must be one of the following:
* `gini`
* `entropy`
Strategy used to choose the split at each node.
Supported strategies are "best" to choose the best split and "random" to choose the best random split.
Values must be one of the following:
* `best`
* `random`
Maximum depth of the tree.
The maximum depth of the tree. If null, then nodes are expanded until all leaves are pure
or until all leaves contain less than min\_samples\_split samples.
Values must be in the following range:
```javascript theme={null}
1 < max_depth < inf
```
Minimum number of samples required to split an internal node.
The minimum number of samples required to split an internal node. If int, then consider min\_samples\_split
as the minimum *count*. If float, then min\_samples\_split is a *fraction* and `ceil(min_samples_split * n_samples)`
are the minimum number of samples for each split.
Minimum number of samples required to be at a leaf node.
The minimum number of samples required to be at a leaf node. A split point at any depth will only be
considered if it leaves at least `min_samples_leaf` training samples in each of the left and right branches.
This may have the effect of smoothing the model, especially in regression.
If int, then consider `min_samples_leaf` as the minimum *count*. If float, then `min_samples_leaf` is
a *fraction* and `ceil(min_samples_leaf * n_samples)` are the minimum number of samples for each node.
Grow a tree with `max_leaf_nodes` in best-first fashion.
Best nodes are defined as relative reduction in impurity. If null then unlimited number of leaf nodes.
Number of features to consider when looking for the best split.
The number of features to consider when looking for the best split:
* If int, then consider `max_features` features at each split.
* If float, then `max_features` is a *fraction* and `int(max_features * n_features)` features are considered at each split.
* If "auto", then `max_features=sqrt(n_features)`.
* If "sqrt", then `max_features=sqrt(n_features)`.
* If "log2", then `max_features=log2(n_features)`.
* If null, then `max_features=n_features`.
Note: the search for a split does not stop until at least one valid partition of the node samples is found,
even if it requires to effectively inspect more than `max_features` features.
Complexity parameter used for Minimal Cost-Complexity Pruning.
Minimal Cost-Complexity Pruning recursively finds the node with the "weakest link". The weakest link is
characterized by an effective alpha, where the nodes with the smallest effective alpha are pruned first.
As alpha increases, more of the tree is pruned, which increases the total impurity of its leaves.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ ccp_alpha < inf
```
Controls the randomness of the estimator.
The features are always randomly permuted at each split, even if splitter is set to "best". When
max\_features \< n\_features, the algorithm will select max\_features at random at each split before
finding the best split among them. But the best found split may vary across different runs, even if
max\_features=n\_features. That is the case, if the improvement of the criterion is identical for several
splits and one split has to be selected at random. To obtain a deterministic behaviour during fitting,
random\_state has to be fixed to an integer.
How to weigh each class of the target variable.
If `null`, all classes will have a weight of one. The "balanced" mode uses the values of the target y to
automatically adjust weights inversely proportional to class frequencies in the input data
as `n_samples / (n_classes * np.bincount(y))`.
Values must be one of the following:
* `balanced`
Configure model validation.
Allows evaluation of model performance via [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html)
using custom metrics. If not specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_splits < inf
```
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [k-fold cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#k-fold)
to split the dataset. E.g. if `n_splits` is 5, the dataset will be split into 5 equal-sized parts.
For five iterations four parts will then be used for training and the remaining part for testing.
If `test_size` is a number between 0 and 1, in contrast, validation is done using a
[shuffle-split](https://scikit-learn.org/stable/modules/cross_validation.html#shufflesplit)
approach. Here, instead of splitting the data into `n_splits` equal parts up front, in each iteration
we randomize the data and sample a proportion equal to `test_size` to use for evaluation and the remaining
rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
Whether to split the data by time.
Most recent data will be used for testing and previous data for training. Assumes data is passed
already sorted ascending by time.
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
Each item in array.
Values must be one of the following:
`explained_variance` `neg_mean_absolute_error` `neg_median_absolute_error` `neg_mean_squared_error` `neg_root_mean_squared_error` `r2`
Configure hypertuning.
Configures the optimization of model hyper-parameters via cross-validated grid- or randomized search.
Which search strategy to use for optimization.
Grid search explores all possible combinations of parameters specified in `params`.
Randomized search, on the other hand, randomly samples `iterations` parameter combinations
from the distributions specified in `params`.
Values must be one of the following:
* `grid`
* `random`
How many randomly sampled parameter combinations to test in randomized search.
Values must be in the following range:
```javascript theme={null}
1 < iterations < inf
```
Configure model validation.
Allows evaluation of model performance via [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html)
using custom metrics. If not specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_splits < inf
```
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [k-fold cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#k-fold)
to split the dataset. E.g. if `n_splits` is 5, the dataset will be split into 5 equal-sized parts.
For five iterations four parts will then be used for training and the remaining part for testing.
If `test_size` is a number between 0 and 1, in contrast, validation is done using a
[shuffle-split](https://scikit-learn.org/stable/modules/cross_validation.html#shufflesplit)
approach. Here, instead of splitting the data into `n_splits` equal parts up front, in each iteration
we randomize the data and sample a proportion equal to `test_size` to use for evaluation and the remaining
rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
Whether to split the data by time.
Most recent data will be used for testing and previous data for training. Assumes data is passed
already sorted ascending by time.
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
Each item in array.
Values must be one of the following:
`explained_variance` `neg_mean_absolute_error` `neg_median_absolute_error` `neg_mean_squared_error` `neg_root_mean_squared_error` `r2`
Metric used to select best model.
Values must be one of the following:
`explained_variance` `neg_mean_absolute_error` `neg_median_absolute_error` `neg_mean_squared_error` `neg_root_mean_squared_error` `r2`
The parameter values to explore.
Allows tuning of any and all of the parameters that can be set also as constants in the
"params" attribute.
Keys in this object should be strings identifying parameter names, and values should be
*lists* of values to explore for that parameter. E.g. `"depth": [3, 5, 7]`.
List of depths values to explore.
The maximum depth of the tree.
Values must be in the following range:
```javascript theme={null}
2 ≤ Item ≤ 16
```
List of iterations values to explore.
Number of iterations.
The maximum number of trees that can be built when solving machine learning problems. When using other parameters that limit the number of iterations, the final number of trees may be less than the number specified in this parameter.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item < inf
```
List of values configuring max cardinality for one-hot encoding.
Maximum cardinality of variables to be one-hot encoded.
Use one-hot encoding for all categorical features with a number of different values less than or equal to this value. Other variables will be target-encoded. Note that one-hot encoding is faster than the alternatives, so decreasing this value makes it more likely slower methods will be used. See [CatBoost details](https://catboost.ai/docs/concepts/algorithm-main-stages_cat-to-numberic.html) for further information.
Values must be in the following range:
```javascript theme={null}
2 ≤ Item < inf
```
List of values configuring variable combination complexity.
The maximum number of features that can be combined when transforming categorical variables.
Each resulting combination consists of one or more categorical features and can optionally contain binary features in the following form: “numeric feature > value”.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item ≤ 4
```
List of leaf regularization strengths.
Coefficient at the L2 regularization term of the cost function.
Values must be in the following range:
```javascript theme={null}
0.0 < Item < inf
```
List of border counts.
The number of splits for numerical features.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item ≤ 65535
```
List of random strengths.
The amount of randomness to use for scoring splits.
Use this parameter to avoid overfitting the model. The value multiplies the variance of a random variable (with
zero mean) that is added to the score used to select splits when a tree is grown.
Values must be in the following range:
```javascript theme={null}
0 < Item < inf
```
List of criterion values to explore.
Function to measure the quality of a split.
Supported criteria are "gini" for the Gini impurity and "entropy" for the information gain.
Values must be one of the following:
* `gini`
* `entropy`
List of splitter values to explore.
Strategy used to choose the split at each node.
Supported strategies are "best" to choose the best split and "random" to choose the best random split.
Values must be one of the following:
* `best`
* `random`
List of max\_depth values to explore.
Maximum depth of the tree.
The maximum depth of the tree. If null, then nodes are expanded until all leaves are pure
or until all leaves contain less than min\_samples\_split samples.
Values must be in the following range:
```javascript theme={null}
1 < Item < inf
```
List of min\_samples\_split values to explore.
Minimum number of samples required to split an internal node.
The minimum number of samples required to split an internal node. If int, then consider min\_samples\_split
as the minimum *count*. If float, then min\_samples\_split is a *fraction* and `ceil(min_samples_split * n_samples)`
are the minimum number of samples for each split.
List of min\_samples\_leaf values to explore.
Minimum number of samples required to be at a leaf node.
The minimum number of samples required to be at a leaf node. A split point at any depth will only be
considered if it leaves at least `min_samples_leaf` training samples in each of the left and right branches.
This may have the effect of smoothing the model, especially in regression.
If int, then consider `min_samples_leaf` as the minimum *count*. If float, then `min_samples_leaf` is
a *fraction* and `ceil(min_samples_leaf * n_samples)` are the minimum number of samples for each node.
List of max\_leaf\_nodes values to explore.
Grow a tree with `max_leaf_nodes` in best-first fashion.
Best nodes are defined as relative reduction in impurity. If null then unlimited number of leaf nodes.
List of max\_features values to explore.
Number of features to consider when looking for the best split.
The number of features to consider when looking for the best split:
* If int, then consider `max_features` features at each split.
* If float, then `max_features` is a *fraction* and `int(max_features * n_features)` features are considered at each split.
* If "auto", then `max_features=sqrt(n_features)`.
* If "sqrt", then `max_features=sqrt(n_features)`.
* If "log2", then `max_features=log2(n_features)`.
* If null, then `max_features=n_features`.
Note: the search for a split does not stop until at least one valid partition of the node samples is found,
even if it requires to effectively inspect more than `max_features` features.
List of ccp\_alpha values to explore.
Complexity parameter used for Minimal Cost-Complexity Pruning.
Minimal Cost-Complexity Pruning recursively finds the node with the "weakest link". The weakest link is
characterized by an effective alpha, where the nodes with the smallest effective alpha are pruned first.
As alpha increases, more of the tree is pruned, which increases the total impurity of its leaves.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ Item < inf
```
Seed for random number generator ensuring reproducibility.
Values must be in the following range:
```javascript theme={null}
0 ≤ seed < inf
```
Train a [RandomForest Regressor](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html).
A versatile ensemble learning method that constructs multiple decision trees during training
and outputs the mean prediction of the individual trees for regression tasks,
enhancing predictive accuracy and controlling over-fitting.
It operates by building numerous decision trees at training time and outputting the average prediction of these trees for regression,
making it highly effective for predictive tasks involving continuous variables. RandomForestRegressor is an ensemble of decision trees designed for regression.
It improves prediction accuracy by randomly selecting subsets of the training data, fitting a decision tree to each subset,
and averaging the predictions. This methodology effectively reduces variance and helps prevent overfitting.
Target variable (labels).
Name of the column that contains your target values (labels).
Importance of each feature in the model.
Whether and how to measure each feature's contribution to the model's predictions. The higher the value,
the more important the feature was in the model. Only relative values are meaningful, i.e. the importance
of a feature relative to other features in the model.
Also note that feature importance is usually meaningful only for models that fit the data well.
The default (`null`, `true` or `"native"`) uses the classifier's native feature importance measure, e.g.
[prediction-value-change](https://catboost.ai/en/docs/concepts/fstr#regular-feature-importance) in the case
of Catboost, [Gini importance](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier.feature_importances_)
in the case of scikit-learn's DecisionTreeClassifier, and the mean of absolute coefficients in the case of
logistic regression.
When set to `"permutation"`, uses [permutation importance](https://scikit-learn.org/stable/modules/permutation_importance.html),
i.e. measures the decrease in model score when a single feature's values are randomly shuffled. This is
considerably slower than native feature importance (the model needs to be evaluated an additional k\*n times,
where k is the number of features and n the number of repetitions to average over). On the positive side it is
model-agnostic and doesn't suffer from bias towards high cardinality features (like some tree-based feature
importances). On the negative side, it can be sensitive to strongly correlated features, as the unshuffled
correlated variable is still available to the model when shuffling the original variable.
When set to `false`, no feature importance will be calculated.
Values must be one of the following:
* `True`
* `False`
* `native`
* `permutation`
* `null`
Toggle encoding of feature columns.
When enabled, Graphext will auto-convert any column types to the numeric type before
fitting the model. How this conversion is done can be configured using the `feature_encoder`
option below.
If disabled, any model trained in this step will assume that input data
is already in an appropriate format (e.g. numerical and not containing any missing values).
Configures encoding of feature columns.
By default (`null`), Graphext chooses automatically how to convert any column types the model
may not understand natively to a numeric type.
A configuration object can be passed instead to overwrite specific parameter values with respect
to their default values.
Numeric encoder.
Configures encoding of numeric features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Further parameters passed to the `scaler` function.
Details depend no the particular scaler used.
Boolean encoder.
Configures encoding of boolean features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Ordinal encoder.
Configures encoding of categorical features that have a natural order.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Category encoder.
May contain either a single configuration for all categorical variables, or two different configurations
for low- and high-cardinality variables. For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for categories with fewer than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Multilabel encoder.
Configures encoding of multivalued categorical features (variable length lists of categories,
or the semantic type `list[category]` for short). May contain either a single configuration for
all multilabel variables, or two different configurations for low- and high-cardinality variables.
For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for mulitabel columns with fewer than `cardinality_threshold` unique categories/labels.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Datetime encoder.
Configures encoding of datetime (timestamp) features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
A list of numerical components to extract.
Will create one numeric column for each component.
Each item in array.
Values must be one of the following:
`day` `dayofweek` `dayofyear` `hour` `minute` `month` `quarter` `season` `second` `week` `weekday` `weekofyear` `year`
A list of cyclical time features to extract.
"Cycles" are numerical transformations of features that should be represented on a circle. E.g. months,
ranging from 1 to 12, should be arranged such that 12 and 1 are next to each other, rather than on
opposite ends of a linear scale. We represent such cyclical time features on a circle by creating two
columns for each original feature: the sin and cos of the numerical feature after appropriate scaling.
Each item in array.
Values must be one of the following:
* `day`
* `dayofweek`
* `dayofyear`
* `hour`
* `month`
Whether to include the epoch as new feature (seconds since 01/01/1970).
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Embedding/vector encoder.
Configures encoding of multivalued numerical features (variable length lists of numbers, i.e. vectors, or the semantic type `list[number]` for short).
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Text encoder.
Configures encoding of text (natural language) features. Currently only allows
[Tf-Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) embeddings to represent texts. If you wish
to use other embeddings, e.g. semantic, Word2Vec etc., transform your text column first using
another step, and then use that result instead of the original texts.
Texts are *excluded* by default from the overall encoding of the dataset. See parameter
`include_text_features` below to active it.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Parameters to be passed to the text encoder (Tf-Idf parameters only for now).
See [scikit-learn's documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)
for detailed parameters and their explanation.
How many output features to generate.
The resulting Tf-Idf vectors will be reduced to these many dimensions (columns) using scikit-learn's
[truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
This performs a kind of [latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
By default we will reduce to 200 components.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_components ≤ 1024
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Whether to include or ignore text columns during the processing of input data.
Enabling this will convert texts to their TfIdf representation. Each text will be
converted to an N-dimensional vector in which each component measures the relative
"over-representation" of a specific word (or n-gram) relative to its overall
frequency in the whole dataset. This is disabled by default because it will
often be better to convert texts explicitly using a previous step, such as
`embed_text` or `embed_text_with_model`.
RandomForest configuration parameters.
You can check the official documentation for more details about RandomForest's parameters [here](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html).
The number of trees in the forest.
The number of trees in the forest. A larger number of trees increases the performance but also the computational cost.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_estimators < inf
```
The function to measure the quality of a split.
The function to measure the quality of a split. Supported criteria are "gini" for the Gini impurity and "entropy" for the information gain.
Values must be one of the following:
* `gini`
* `entropy`
The maximum depth of the tree.
The maximum depth of the tree. If `null`, then nodes are expanded until all leaves are pure or until all leaves contain less than min\_samples\_split samples.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_depth < inf
```
How to weigh each class of the target variable.
If `null`, all classes will have a weight of one. The "balanced" mode uses the values of the target y to
automatically adjust weights inversely proportional to class frequencies in the input data
as `n_samples / (n_classes * np.bincount(y))`.
The “balanced\_subsample” mode is the same as “balanced” except that weights are computed based on the bootstrap
sample for every tree grown.
Values must be one of the following:
* `balanced`
* `balanced_subsample`
* `None`
The minimum number of samples required to split an internal node.
The minimum number of samples required to split an internal node. A split point at any depth will only be considered if it leaves at least `min_samples_split` training samples in each of the left and right branches.
Values must be in the following range:
```javascript theme={null}
2 ≤ min_samples_split < inf
```
The minimum number of samples required to be at a leaf node.
A split point at any depth will only be considered if it leaves at least `min_samples_leaf` training samples in each of the left and right branches. This may have the effect of smoothing the model.
Values must be in the following range:
```javascript theme={null}
1 ≤ min_samples_leaf < inf
```
The minimum weighted fraction of the sum total of weights required to be at a leaf node.
The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node. Samples have equal weight when sample\_weight is not provided.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ min_weight_fraction_leaf ≤ 0.5
```
The number of features to consider when looking for the best split.
The number of features to consider when looking for the best split. If “auto”, then `max_features=sqrt(n_features)`. If `null`, then `max_features=n_features`.
Grow trees with max\_leaf\_nodes in best-first fashion.
Grow trees with `max_leaf_nodes` in best-first fashion. Best nodes are defined as relative reduction in impurity. If `null` then unlimited number of leaf nodes.
A node will be split if this split induces a decrease of the impurity greater than or equal to this value.
A node will be split if this split induces a decrease of the impurity greater than or equal to this value. This may have the effect of smoothing the model, especially in regression.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ min_impurity_decrease < inf
```
Whether bootstrap samples are used when building trees.
If `true`, bootstrap samples are used when building trees. If `false`, the whole dataset is used to build each tree.
Controls both the randomness of the bootstrapping of the samples used when building trees and the sampling of the features to consider when looking for the best split at each node.
If bootstrap is True, the number of samples to draw from X to train each base estimator.
If bootstrap is True, the number of samples to draw from X to train each base estimator. If `null` (default), then draw `X.shape[0]` samples.
Complexity parameter used for Minimal Cost-Complexity Pruning.
Complexity parameter used for Minimal Cost-Complexity Pruning. The subtree with the largest cost complexity that is smaller than `ccp_alpha` will be chosen. By default, no pruning is performed.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ ccp_alpha < inf
```
The number of jobs to run in parallel for both `fit` and `predict`.
The number of jobs to run in parallel for both `fit` and `predict`. `-1` means using all processors.
Controls the verbosity when fitting and predicting.
Controls the verbosity when fitting and predicting.
Values must be in the following range:
```javascript theme={null}
0 ≤ verbose < inf
```
Configure model validation.
Allows evaluation of model performance via [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html)
using custom metrics. If not specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_splits < inf
```
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [k-fold cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#k-fold)
to split the dataset. E.g. if `n_splits` is 5, the dataset will be split into 5 equal-sized parts.
For five iterations four parts will then be used for training and the remaining part for testing.
If `test_size` is a number between 0 and 1, in contrast, validation is done using a
[shuffle-split](https://scikit-learn.org/stable/modules/cross_validation.html#shufflesplit)
approach. Here, instead of splitting the data into `n_splits` equal parts up front, in each iteration
we randomize the data and sample a proportion equal to `test_size` to use for evaluation and the remaining
rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
Whether to split the data by time.
Most recent data will be used for testing and previous data for training. Assumes data is passed
already sorted ascending by time.
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
Each item in array.
Values must be one of the following:
`explained_variance` `neg_mean_absolute_error` `neg_median_absolute_error` `neg_mean_squared_error` `neg_root_mean_squared_error` `r2`
Seed for random number generator ensuring reproducibility.
Values must be in the following range:
```javascript theme={null}
0 ≤ seed < inf
```
Train an [ExtraTrees Regressor](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.ExtraTreesRegressor.html).
Uses randomized trees and averaging on dataset sub-samples to enhance prediction and reduce overfitting.
The ExtraTreesRegressor is an ensemble learning method fundamentally similar to a random forest.
It fits a number of randomized decision trees on various sub-samples of the dataset and uses
averaging to improve the predictive accuracy and control over-fitting.
The main difference from the random forest is in the way it splits nodes, which is random in ExtraTrees,
leading to more diversified trees and thus, a more robust model against overfitting on the training data.
Target variable (labels).
Name of the column that contains your target values (labels).
Importance of each feature in the model.
Whether and how to measure each feature's contribution to the model's predictions. The higher the value,
the more important the feature was in the model. Only relative values are meaningful, i.e. the importance
of a feature relative to other features in the model.
Also note that feature importance is usually meaningful only for models that fit the data well.
The default (`null`, `true` or `"native"`) uses the classifier's native feature importance measure, e.g.
[prediction-value-change](https://catboost.ai/en/docs/concepts/fstr#regular-feature-importance) in the case
of Catboost, [Gini importance](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier.feature_importances_)
in the case of scikit-learn's DecisionTreeClassifier, and the mean of absolute coefficients in the case of
logistic regression.
When set to `"permutation"`, uses [permutation importance](https://scikit-learn.org/stable/modules/permutation_importance.html),
i.e. measures the decrease in model score when a single feature's values are randomly shuffled. This is
considerably slower than native feature importance (the model needs to be evaluated an additional k\*n times,
where k is the number of features and n the number of repetitions to average over). On the positive side it is
model-agnostic and doesn't suffer from bias towards high cardinality features (like some tree-based feature
importances). On the negative side, it can be sensitive to strongly correlated features, as the unshuffled
correlated variable is still available to the model when shuffling the original variable.
When set to `false`, no feature importance will be calculated.
Values must be one of the following:
* `True`
* `False`
* `native`
* `permutation`
* `null`
Toggle encoding of feature columns.
When enabled, Graphext will auto-convert any column types to the numeric type before
fitting the model. How this conversion is done can be configured using the `feature_encoder`
option below.
If disabled, any model trained in this step will assume that input data
is already in an appropriate format (e.g. numerical and not containing any missing values).
Configures encoding of feature columns.
By default (`null`), Graphext chooses automatically how to convert any column types the model
may not understand natively to a numeric type.
A configuration object can be passed instead to overwrite specific parameter values with respect
to their default values.
Numeric encoder.
Configures encoding of numeric features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Further parameters passed to the `scaler` function.
Details depend no the particular scaler used.
Boolean encoder.
Configures encoding of boolean features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Ordinal encoder.
Configures encoding of categorical features that have a natural order.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Category encoder.
May contain either a single configuration for all categorical variables, or two different configurations
for low- and high-cardinality variables. For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for categories with fewer than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Multilabel encoder.
Configures encoding of multivalued categorical features (variable length lists of categories,
or the semantic type `list[category]` for short). May contain either a single configuration for
all multilabel variables, or two different configurations for low- and high-cardinality variables.
For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for mulitabel columns with fewer than `cardinality_threshold` unique categories/labels.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Datetime encoder.
Configures encoding of datetime (timestamp) features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
A list of numerical components to extract.
Will create one numeric column for each component.
Each item in array.
Values must be one of the following:
`day` `dayofweek` `dayofyear` `hour` `minute` `month` `quarter` `season` `second` `week` `weekday` `weekofyear` `year`
A list of cyclical time features to extract.
"Cycles" are numerical transformations of features that should be represented on a circle. E.g. months,
ranging from 1 to 12, should be arranged such that 12 and 1 are next to each other, rather than on
opposite ends of a linear scale. We represent such cyclical time features on a circle by creating two
columns for each original feature: the sin and cos of the numerical feature after appropriate scaling.
Each item in array.
Values must be one of the following:
* `day`
* `dayofweek`
* `dayofyear`
* `hour`
* `month`
Whether to include the epoch as new feature (seconds since 01/01/1970).
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Embedding/vector encoder.
Configures encoding of multivalued numerical features (variable length lists of numbers, i.e. vectors, or the semantic type `list[number]` for short).
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Text encoder.
Configures encoding of text (natural language) features. Currently only allows
[Tf-Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) embeddings to represent texts. If you wish
to use other embeddings, e.g. semantic, Word2Vec etc., transform your text column first using
another step, and then use that result instead of the original texts.
Texts are *excluded* by default from the overall encoding of the dataset. See parameter
`include_text_features` below to active it.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Parameters to be passed to the text encoder (Tf-Idf parameters only for now).
See [scikit-learn's documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)
for detailed parameters and their explanation.
How many output features to generate.
The resulting Tf-Idf vectors will be reduced to these many dimensions (columns) using scikit-learn's
[truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
This performs a kind of [latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
By default we will reduce to 200 components.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_components ≤ 1024
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Whether to include or ignore text columns during the processing of input data.
Enabling this will convert texts to their TfIdf representation. Each text will be
converted to an N-dimensional vector in which each component measures the relative
"over-representation" of a specific word (or n-gram) relative to its overall
frequency in the whole dataset. This is disabled by default because it will
often be better to convert texts explicitly using a previous step, such as
`embed_text` or `embed_text_with_model`.
RandomForest configuration parameters.
You can check the official documentation for more details about RandomForest's parameters [here](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html).
The number of trees in the forest.
The number of trees in the forest. A larger number of trees increases the performance but also the computational cost.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_estimators < inf
```
The function to measure the quality of a split.
The function to measure the quality of a split. Supported criteria are "gini" for the Gini impurity and "entropy" for the information gain.
Values must be one of the following:
* `gini`
* `entropy`
The maximum depth of the tree.
The maximum depth of the tree. If `null`, then nodes are expanded until all leaves are pure or until all leaves contain less than min\_samples\_split samples.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_depth < inf
```
How to weigh each class of the target variable.
If `null`, all classes will have a weight of one. The "balanced" mode uses the values of the target y to
automatically adjust weights inversely proportional to class frequencies in the input data
as `n_samples / (n_classes * np.bincount(y))`.
The “balanced\_subsample” mode is the same as “balanced” except that weights are computed based on the bootstrap
sample for every tree grown.
Values must be one of the following:
* `balanced`
* `balanced_subsample`
* `None`
The minimum number of samples required to split an internal node.
The minimum number of samples required to split an internal node. A split point at any depth will only be considered if it leaves at least `min_samples_split` training samples in each of the left and right branches.
Values must be in the following range:
```javascript theme={null}
2 ≤ min_samples_split < inf
```
The minimum number of samples required to be at a leaf node.
A split point at any depth will only be considered if it leaves at least `min_samples_leaf` training samples in each of the left and right branches. This may have the effect of smoothing the model.
Values must be in the following range:
```javascript theme={null}
1 ≤ min_samples_leaf < inf
```
The minimum weighted fraction of the sum total of weights required to be at a leaf node.
The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node. Samples have equal weight when sample\_weight is not provided.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ min_weight_fraction_leaf ≤ 0.5
```
The number of features to consider when looking for the best split.
The number of features to consider when looking for the best split. If “auto”, then `max_features=sqrt(n_features)`. If `null`, then `max_features=n_features`.
Grow trees with max\_leaf\_nodes in best-first fashion.
Grow trees with `max_leaf_nodes` in best-first fashion. Best nodes are defined as relative reduction in impurity. If `null` then unlimited number of leaf nodes.
A node will be split if this split induces a decrease of the impurity greater than or equal to this value.
A node will be split if this split induces a decrease of the impurity greater than or equal to this value. This may have the effect of smoothing the model, especially in regression.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ min_impurity_decrease < inf
```
Whether bootstrap samples are used when building trees.
If `true`, bootstrap samples are used when building trees. If `false`, the whole dataset is used to build each tree.
Controls both the randomness of the bootstrapping of the samples used when building trees and the sampling of the features to consider when looking for the best split at each node.
If bootstrap is True, the number of samples to draw from X to train each base estimator.
If bootstrap is True, the number of samples to draw from X to train each base estimator. If `null` (default), then draw `X.shape[0]` samples.
Complexity parameter used for Minimal Cost-Complexity Pruning.
Complexity parameter used for Minimal Cost-Complexity Pruning. The subtree with the largest cost complexity that is smaller than `ccp_alpha` will be chosen. By default, no pruning is performed.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ ccp_alpha < inf
```
The number of jobs to run in parallel for both `fit` and `predict`.
The number of jobs to run in parallel for both `fit` and `predict`. `-1` means using all processors.
Controls the verbosity when fitting and predicting.
Controls the verbosity when fitting and predicting.
Values must be in the following range:
```javascript theme={null}
0 ≤ verbose < inf
```
Configure model validation.
Allows evaluation of model performance via [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html)
using custom metrics. If not specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_splits < inf
```
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [k-fold cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#k-fold)
to split the dataset. E.g. if `n_splits` is 5, the dataset will be split into 5 equal-sized parts.
For five iterations four parts will then be used for training and the remaining part for testing.
If `test_size` is a number between 0 and 1, in contrast, validation is done using a
[shuffle-split](https://scikit-learn.org/stable/modules/cross_validation.html#shufflesplit)
approach. Here, instead of splitting the data into `n_splits` equal parts up front, in each iteration
we randomize the data and sample a proportion equal to `test_size` to use for evaluation and the remaining
rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
Whether to split the data by time.
Most recent data will be used for testing and previous data for training. Assumes data is passed
already sorted ascending by time.
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
Each item in array.
Values must be one of the following:
`explained_variance` `neg_mean_absolute_error` `neg_median_absolute_error` `neg_mean_squared_error` `neg_root_mean_squared_error` `r2`
Seed for random number generator ensuring reproducibility.
Values must be in the following range:
```javascript theme={null}
0 ≤ seed < inf
```
# train_survival
Source: https://docs.graphext.com/api-docs/analyse/train_and_predict/train_survival
Train and store a survival model to be loaded at a later point for prediction.
Trains a survival model using the Cox Proportional Hazard model.
The output will always be a new column with the trained model's predictions on the training data,
as well as a saved and named model file that can be used in other projects for prediction of new data.
## Usage
The following examples show how the step can be used in a recipe.
Train a Cox Proportional Hazard survival model
```stan theme={null}
train_survival(ds, {"target": ["event_observed", "duration"]}) -> (ds.predicted_survival, "my-survival-model")
```
Train a CoxPH model with penalization and median survival time prediction
```stan theme={null}
train_survival(ds, {"target": ["event_observed", "time_to_event"], "predictions": {"kind": "median"}, "params": {"penalizer": 0.1}}) -> (ds.predicted_survival, "my-survival-model")
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
train_survival(ds: dataset, {
"param": value,
...
}) -> (predicted: number, model: model_survival[ds])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Should contain the target columns (see `target` parameter below) and the feature columns you wish to use in the model.
Name for output column containing model predictions.
Zip file containing the trained model and associated information.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Kind of survival model to train.
"CoxPH" trains a [lifelines Cox Proportional Hazard model](https://lifelines.readthedocs.io/en/latest/Survival%20Regression.html#cox-s-proportional-hazard-model).
Target variables.
Two names, exactly, corresponding to the target columns that contain in the following order:
1. whether the event was observed (boolean) and
2. the time (duration) to event or censoring (number).
Configure the kind of predictions to return.
Kind of prediction.
`median` returns the median survival time. `percentile` returns the survival time at
the given percentile. `expectation` returns the expected survival time.
`survival_function` returns the whole survival function (one series per sample).
Values must be one of the following:
* `median`
* `percentile`
* `expectation`
* `survival_function`
Percentile when `kind` is set to `percentile`
Values must be in the following range:
```javascript theme={null}
0 ≤ percentile ≤ 1
```
Points in time to predict.
Configures at which points to predict when `kind` is set to `survival_function`.
Either an explicit array of durations, or an object specifying a duration step size and
maximum duration.
Array of times/durations.
Will predict the survival function at each of the durations. E.g. `[1, 2, 3, 4, 5]`.
Each item in array.
Step size.
The step size in the enumeration of durations. E.g. `1` will predict the survival
function at each integer duration.
Values must be in the following range:
```javascript theme={null}
0 < step < inf
```
Maximum duration.
If not provided, or `null`, the maximum duration in the dataset is used.
Model parameters.
Level in the confidence intervals.
Penalizer strength.
Attach an L2 penalizer to the size of the coefficients during regression.
This improves stability of the estimates and controls for high correlation between covariates.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ penalizer < inf
```
L1 vs L2 penalty ratio.
Specify what ratio to assign to a L1 vs L2 penalty (ridge vs lasso). Same as scikit-learn
convention.
Values must be in the following range:
```javascript theme={null}
0.0 ≤ l1_ratio ≤ 1.0
```
Columns to use in stratification.
This is useful if a categorical covariate does not obey the proportional hazard assumption.
Each item in array.
How the fitter should estimate the baseline.
Values must be one of the following:
* `breslow`
* `spline`
* `piecewise`
# aggregate
Source: https://docs.graphext.com/api-docs/prepare/aggregate/aggregate
Group and aggregate a dataset using any of a number of predefined functions.
After optionally sorting the dataset, it is grouped by the unique values (or combinations of unique values)
in one or more columns. Each group's rows are then aggregated using one or more predefined functions. A new
dataset is thus created containing one column per selected aggregation function, and one row for each unique
group.
## Usage
The following example shows how the step can be used in a recipe.
Given an online retail dataset `products`, where rows represent items with id `product_id`, and which have
been added to a shopping basket at time `time_added`, we can aggregate these items into a new dataset `baskets`
containing one row per basket. The following configuration calculates this aggregation, creating a new dataset
with three columns:
* `products`: a list of all items in a given basket, preserving the order they were added
* `size`: the number of items in the basket
* `total`: the total value of the basket
```stan theme={null}
aggregate(products, {
"by": "order_id",
"presort": {
"columns": "time_added"
},
"aggregations": {
"product_id": {
"products": {"func": "list"},
"size": {"func": "count"}
},
"item_total": {
"total": {"func": "sum"}}
}
}) -> (baskets)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
aggregate(ds_in: dataset, {
"param": value,
...
}) -> (ds_out: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A dataset to group and aggregate.
The result of the aggregation.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Grouping column(s).
The name(s) of column(s) whose unique values define the groups to aggregate.
Each item in array.
* order\_id
* \['weekday', 'hour']
Pre-aggregation row sorting.
Sort the dataset rows before aggregating, e.g. when in a particular aggregation function (such as `list`) the
encountered order is important.
The sort column name(s).
These column(s) will be used to sort the dataset before aggregating (if multiple, in specified order).
null.
string.
array.
Each item in array.
* date\_added
* \['lastname', 'firstname']
Whether to sort in ascending order (or in descending order if false).
* With a single column for sorting:
```json theme={null}
"presort": {
"columns": "date_added",
"ascending": true
}
```
Definition of desired aggregations.
A dictionary mapping original columns to new aggregated columns, specifying an aggregation function for each.
*Aggregations* are functions that reduce all the values in a particular column of a single group to a single summary value of that group.
E.g. a `sum` aggregation of column A calculates a single total by adding up all the values in A belonging to each group.
Possible aggregations functions accepted as `func` parameters are:
* `n`, `size` or `count`: calculate number of rows in group
* `sum`: sum total of values
* `mean`: take mean of values
* `max`: take max of values
* `min`: take min of values
* `mode`: find most frequent value (returns first mode if multiple exist)
* `first`: take first item found
* `last`: take last item found
* `unique`: collect a list of unique values
* `n_unique`: count the number of unique values
* `list`: collect a list of all values
* `concatenate`: convert all values to text and concatenate them into one long text
* `concat_lists`: concatenate lists in all rows into a single larger list
* `count_where`: number of rows in which the column matches a value, needs parameter `value` with the value that you want to count
* `percent_where`: percentage of the column where the column matches a value, needs parameter `value` with the value that you want to count
Note that in the case of `count_where` and `percent_where` an additional `value` parameter is required.
One item per input column.
Each key should be the name of an input column, and each value an object defining one or more aggregations for that column.
An individual aggregation consists of the name of a desired output column, mapped to a specific aggregation function.
For example:
```json theme={null}
{
"input_col": {
"output_col": {"func": "sum"}
}
}
```
Object defining how to aggregate a single output column.
Needs at least the `"func"` parameter. If the aggregation function accepts further arguments,
like the `"value"` parameter in case of `count_where` and `percent_where`, these need to be provided also.
For example:
```json theme={null}
{
"output_col": {"func": "count_where", "value": 2}
}
```
Aggregation function.
Values must be one of the following:
`n` `size` `count` `sum` `mean` `n_unique` `count_where` `percent_where` `concatenate` `max` `min` `first` `last` `mode` `concat_lists` `unique` `list`
* Including an aggregation function with additional parameters:
```json theme={null}
{
"product_id": {
"products": {"func": "list"},
"size": {"func": "count"}
},
"item_total": {
"total": {"func": "sum"},
},
"item_category": {
"num_food_items": {"func": "count_where", "value": "food"}
}
}
```
Whether to ignore missing values (NaNs) in group columns.
If `false` (default), missing values (NaNs) will be grouped together in their own group. Otherwise, rows
containing NaNs in the group column will be ignored.
Whether to sort groups by values in the grouping columns.
This doesn't affect sorting of rows *within* groups, which is always maintained (and may depend on the
`presort` parameter), but only the ordering *amongst* groups. If the order of groups is not important,
leaving this off will usually result in faster execution (`false` by default) .
Enforce use of Pandas aggregation.
Normally, depending on dataset size, the step will automatically switch between Pandas and Dask aggregation,
preferring whichever represents a better trade-off between execution-time and memory usage. For very
large datasets, Dask is the only viable method, but Dask has limitations when it comes to sorting.
For intermediate dataset sizes, and if you need to sort the dataset before aggregation on more than a single
column, you can try enforcing the use of Pandas if otherwise you see warning or errors related to sorting.
# aggregate_list_items
Source: https://docs.graphext.com/api-docs/prepare/aggregate/aggregate_list_items
Group a dataset by elements in a column of lists and aggregate remaining columns using one or more predefined functions.
This is essentially `aggregate` after "exploding" a column of lists such that each list item has its
own row. By default the step produces one row per unique list item, and two columns: the `count` of how many
times each list item was encountered, and a column `rows` recording the row numbers of the lists in which the
element was found (\[1,3,7] would mean an item was present in the lists of rows 1, 3 and 7). In addition,
predefined functions can be used to add further aggregations of the grouped input dataset.
For example, if a dataset contains texts already separated into lists of individual words, this step will create a
new dataset containing one row per word, a column containing each word's frequency (count) across all texts, and
another column of lists indicating in which rows the word was found.
Optionally, if a grouping column is specified using the `"by"` parameter, otherwise identical items belonging to
different groups will be counted separately. If the dataset contains texts in different languages, for example, one
may not want to group all occurences of the same word together, irrespective of language. The word "angel"
in German signifies a fishing rod, for example, "any" in Catalan means "year", and the Italian word "burro" means
"butter" while in Spanish it refers to "donkey". Using language as the grouping column would preserve the word in each
language as a separate group.
## Usage
The following example shows how the step can be used in a recipe.
The following example performs a simple word-count, returning a new dataset with one row per word and
each word's frequency in the "count" column. The aggregation will be performed separately for each language.
Also, for each word, a custom aggregation collects the dates of the texts in which the word was mentioned:
```stan theme={null}
aggregate_list_items(ds_in, {
"split_column": "words",
"by": "language",
"unique_rows": true,
"aggregations": {
"text_publication_date": {
"mention_dates": {"func": "list"},
}
}
}) -> (ds_out)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
aggregate_list_items(ds_in: dataset, {
"param": value,
...
}) -> (ds_out: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An input dataset containing at least one column with lists of elements to group.
The result of the aggregation. Contains one row per unique element in original column of lists.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Name of column containing the lists to be split and grouped.
Optional grouping column to use for item counting and aggregation.
Count unique occurences only.
Whether to collect in the output column "rows" only the unique rows each item appeared in,
or all rows (duplicate row IDs if item appeared more than once in a single row).
Row IDs as strings.
Output occurrence of items in rows as lists of strings (categorical) rather than lists of row numbers.
Definition of additional aggregations.
A dictionary mapping original columns to new aggregated columns, specifying an aggregation function for each.
*Aggregations* are functions that reduce all the values in a particular column of a single group to a single summary value of that group.
E.g. a `sum` aggregation of column A calculates a single total by adding up all the values in A belonging to each group.
Possible aggregations functions accepted as `func` parameters are:
* `n`, `size` or `count`: calculate number of rows in group
* `sum`: sum total of values
* `mean`: take mean of values
* `max`: take max of values
* `min`: take min of values
* `mode`: find most frequent value (returns first mode if multiple exist)
* `first`: take first item found
* `last`: take last item found
* `unique`: collect a list of unique values
* `n_unique`: count the number of unique values
* `list`: collect a list of all values
* `concatenate`: convert all values to text and concatenate them into one long text
* `concat_lists`: concatenate lists in all rows into a single larger list
* `count_where`: number of rows in which the column matches a value, needs parameter `value` with the value that you want to count
* `percent_where`: percentage of the column where the column matches a value, needs parameter `value` with the value that you want to count
Note that in the case of `count_where` and `percent_where` an additional `value` parameter is required.
One item per input column.
Each key should be the name of an input column, and each value an object defining one or more aggregations for that column.
An individual aggregation consists of the name of a desired output column, mapped to a specific aggregation function.
For example:
```json theme={null}
{
"input_col": {
"output_col": {"func": "sum"}
}
}
```
Object defining how to aggregate a single output column.
Needs at least the `"func"` parameter. If the aggregation function accepts further arguments,
like the `"value"` parameter in case of `count_where` and `percent_where`, these need to be provided also.
For example:
```json theme={null}
{
"output_col": {"func": "count_where", "value": 2}
}
```
Aggregation function.
Values must be one of the following:
`n` `size` `count` `sum` `mean` `n_unique` `count_where` `percent_where` `concatenate` `max` `min` `first` `last` `mode` `concat_lists` `unique` `list`
* Including an aggregation function with additional parameters:
```json theme={null}
{
"product_id": {
"products": {"func": "list"},
"size": {"func": "count"}
},
"item_total": {
"total": {"func": "sum"},
},
"item_category": {
"num_food_items": {"func": "count_where", "value": "food"}
}
}
```
# aggregate_neighbours
Source: https://docs.graphext.com/api-docs/prepare/aggregate/aggregate_neighbours
For each node in a network, group and aggregate over its neighbours.
Using the link columns in the provided dataset (including at least a targets columns containing
lists of target row numbers that each row connects to), for each row calculate requested aggregations
over all its direct (first-degree) neighbours.
Will use the first set of link columns encountered in the datasets metadata.
## Usage
The following example shows how the step can be used in a recipe.
Assuming a dataset `products` where each row represents a supermarket product (having at least a `price` and `aisle` column),
and containing a targets column dataset representing connections between similar products, the following example calculates for
each product
* the average price of similar products
* the percentage of similar products assigned to aisles "produce", "deli" and "drinks"
```stan theme={null}
aggregate_neighbours(products, {
"aggregations": {
"price": {
"similar_price_avg": {"func": "mean"}
},
"aisle": {
"similar_pct_produce": {"func": "percent_where", "value": "produce"},
"similar_pct_deli": {"func": "percent_where", "value": "deli"},
"similar_pct_drinks": {"func": "percent_where", "value": "drinks"}
}
}
}) -> (products_agg)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
aggregate_neighbours(ds_in: dataset, {
"param": value,
...
}) -> (ds_out: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A dataset containing the nodes (rows) to group and aggregate, and its corresponding links.
The original dataset plus newly aggregated columns. Will have one column per specified aggregation function
(more than one aggregation can be specified for each original input column).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Pre-aggregation row sorting.
Sort the dataset rows before aggregating, e.g. when in a particular aggregation function (such as `list`) the encountered order is important.
The sort column name(s).
These column(s) will be used to sort the dataset before aggregating (if multiple, in specified order).
E.g. to first sort links by their weight, and if the weight column is called "gx\_weight", use `"gx_weight"`
When `null`, no sorting is applied.
Single column name. Sort by this column.
List of column names. Orders by the first column, then the second, etc.
Each item in array.
* date\_added
* \['lastname', 'firstname']
Whether to sort in ascending order (or in descending order if false).
* For example, to sort first by price, then dimension, and in descending order:
```json theme={null}
{
"columns": ["price", "dimension"],
"ascending": false
}
```
Definition of desired aggregations.
A dictionary mapping original columns to new aggregated columns, specifying an aggregation function for each.
*Aggregations* are functions that reduce all the values in a particular column of a single group to a single summary value of that group.
E.g. a `sum` aggregation of column A calculates a single total by adding up all the values in A belonging to each group.
Possible aggregations functions accepted as `func` parameters are:
* `n`, `size` or `count`: calculate number of rows in group
* `sum`: sum total of values
* `mean`: take mean of values
* `max`: take max of values
* `min`: take min of values
* `mode`: find most frequent value (returns first mode if multiple exist)
* `first`: take first item found
* `last`: take last item found
* `unique`: collect a list of unique values
* `n_unique`: count the number of unique values
* `list`: collect a list of all values
* `concatenate`: convert all values to text and concatenate them into one long text
* `concat_lists`: concatenate lists in all rows into a single larger list
* `count_where`: number of rows in which the column matches a value, needs parameter `value` with the value that you want to count
* `percent_where`: percentage of the column where the column matches a value, needs parameter `value` with the value that you want to count
Note that in the case of `count_where` and `percent_where` an additional `value` parameter is required.
One item per input column.
Each key should be the name of an input column, and each value an object defining one or more aggregations for that column.
An individual aggregation consists of the name of a desired output column, mapped to a specific aggregation function.
For example:
```json theme={null}
{
"input_col": {
"output_col": {"func": "sum"}
}
}
```
Object defining how to aggregate a single output column.
Needs at least the `"func"` parameter. If the aggregation function accepts further arguments,
like the `"value"` parameter in case of `count_where` and `percent_where`, these need to be provided also.
For example:
```json theme={null}
{
"output_col": {"func": "count_where", "value": 2}
}
```
Aggregation function.
Values must be one of the following:
`n` `size` `count` `sum` `mean` `n_unique` `count_where` `percent_where` `concatenate` `max` `min` `first` `last` `mode` `concat_lists` `unique` `list`
* Including an aggregation function with additional parameters:
```json theme={null}
{
"product_id": {
"products": {"func": "list"},
"size": {"func": "count"}
},
"item_total": {
"total": {"func": "sum"},
},
"item_category": {
"num_food_items": {"func": "count_where", "value": "food"}
}
}
```
Whether the links provided should be interpreted as being directed.
*Directed* here meaning that the link A→B (from node A to B) may be different from the link B→A (i.e. they may
have different weight attributes for example). When `"directed": false`, in contrast, i.e. links are *undirected*,
it is assumed that the link A→B is always identical to B→A (i.e. A↔B always). This is usually the case when
links represent a *similarity* between nodes.
# aggregate_tweets_by_author
Source: https://docs.graphext.com/api-docs/prepare/aggregate/aggregate_tweets_by_author
Group a dataset of tweets by author and calculate relevant author statistics.
Works like the generic `aggregate` step, but with a predefined set of aggregation functions. See the `ds_out` argument below
for the columns generated in the resulting dataset.
## Usage
The following examples show how the step can be used in a recipe.
Aggregate tweets by author using standard Twitter column names
```stan theme={null}
aggregate_tweets_by_author(ds) -> (ds_authors)
```
Aggregate with custom column mapping
```stan theme={null}
aggregate_tweets_by_author(ds, {"add_referenced_accounts": false, "column_map": {"user_id": "author_id", "screen_name": "author_handler"}}) -> (ds_authors)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
aggregate_tweets_by_author(ds_in: dataset, {
"param": value,
...
}) -> (ds_out: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A dataset where each row is a tweet.
Result of the aggregation, where each row is a twitter account. It will include for each author up to the following columns,
depending on information present on the original dataset:
* `author_id`: Official Twitter ID
* `tweet_count`: Number of tweets by this author
* `handler`: Official Twitter handle
* `name`: User name
* `pic`: Link to user's profile picture
* `links`: A list of links mentioned by the user
* `dates`: A list of dates of published tweets by this author
* `tweet_ids`: The official Twitter IDs of the tweets published by the author
* `retweets`: The number of retweets received
* `favorites`: The number of favorites received
* `mention_ids`: List of other accounts (IDs) the author has *mentioned*
* `mention_names`: List of other accounts (names) the author has *mentioned*
* `rp_user_ids`: List of other accounts (IDs) the author has *replied* to
* `rp_user_names`: List of other accounts (names) the author has *replied* to
* `mentions`: The count of mentions received
* `replies`: The count of replies received
* `tweet_text`: The text of the author's tweets, concatenated.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Whether to add rows for accounts only "mentioned" in original tweets.
If mentions or replies are recorded in the dataset (in columns `mention_ids`, `mention_names`
and/or `rp_user_id`, `rp_user_name`) will add the corresponding accounts as rows in the result,
even if they didn't have a tweet in the original dataset.
Will add `mentions` and `replies` columns recording how many times the accounts were
mentioned or replied to.
Column Map.
If the names of any of your dataset's columns don't correspond to those we expect
to find in a tweet dataset (e.g. originating in Twitter's own API), you can provide
a mapping of of the sort `{"your_column": "author_id"}`.
The expected column names are `[author_id, author_handler, author_name, author_avatar,
links, date, id, retweets, favorites, mention_ids, mention_names, rp_user_id, rp_user_name
, text]`.
Column name to map.
Values must be one of the following:
`author_id` `author_handler` `author_name` `author_avatar` `links` `date` `id` `retweets` `favorites` `mention_ids` `mention_names` `rp_user_id` `rp_user_name` `text`
# featurize_time_series
Source: https://docs.graphext.com/api-docs/prepare/aggregate/featurize_time_series
Summarizes time series data into aggregate metrics.
Extracts features from time series data for machine learning or analysis.
Supports three feature sets:
* [**catch22**](https://time-series-features.gitbook.io/catch22): 22 time series features, plus optional mean and standard deviation (24 total).
See details about each feature [here](https://time-series-features.gitbook.io/catch22/information-about-catch22/feature-descriptions/feature-overview-table).
* [**tsfeatures**](https://github.com/Nixtla/tsfeatures): Statistical features including trend, seasonality, autocorrelation, etc.
See details about each feature [here](https://cran.r-project.org/web/packages/tsfeatures/vignettes/tsfeatures.html).
* **growth**: Simple, average, compound, and linear growth metrics.
The step takes a dataset with time series data in "tall" format (one row per time point)
or "wide" format (time points in columns), and produces a dataset with the calculated features.
## Usage
The following example shows how the step can be used in a recipe.
To calculate all "growth" metrics:
```stan theme={null}
featurize_time_series(ds, {
"id": "product_id",
"time": "time_added",
"value": "item_total",
"sets": ["growth"]
}) -> (features)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
featurize_time_series(ds: dataset, {
"param": value,
...
}) -> (features: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A dataset containing time series.
A dataset containing time series features.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Column name containing the time series identifier.
Column name containing the timestamps.
Column name containing the values to featurize.
Feature sets to include.
E.g. "catch22" or "tsfeatures". If no individual features are configured using the `features` pararmeter,
all features from the selected set will be computed. If multiple sets are selected, all features from
each set will be computed. If `all` is selected, all features from all sets will be computed.
Values must be one of the following:
* `catch22`
* `tsfeatures`
* `growth`
* `all`
Each item in array.
Values must be one of the following:
* `catch22`
* `tsfeatures`
* `growth`
* `all`
Custom features to compute from each feature set.
Catch22 features to compute.
[See here](https://time-series-features.gitbook.io/catch22/information-about-catch22/feature-descriptions/feature-overview-table)
for detailed information about each possible feature.
Each item should be a name of a Catch22 feature.
Values must be one of the following:
`mode_5` `mode_10` `acf_timescale` `acf_first_min` `ami2` `trev` `high_fluctuation` `stretch_high` `transition_matrix` `periodicity` `embedding_dist` `ami_timescale` `whiten_timescale` `outlier_timing_pos` `outlier_timing_neg` `centroid_freq` `stretch_decreasing` `entropy_pairs` `rs_range` `dfa` `low_freq_power` `forecast_error` `mean` `SD`
TSFeatures features to compute.
[See here](https://cran.r-project.org/web/packages/tsfeatures/vignettes/tsfeatures.html)
for detailed information about each possible feature.
Each item should be the name of a TSFeature feature.
Values must be one of the following:
`acf_features` `arch_stat` `crossing_points` `entropy` `flat_spots` `heterogeneity` `holt_parameters` `lumpiness` `nonlinearity` `pacf_features` `stl_features` `stability` `hw_parameters` `unitroot_kpss` `unitroot_pp` `series_length` `hurst`
Growth features to compute.
The different growth features are calculated as follows, where $x_f$ is the final value, $x_0$
is the initial value, and $n$ is the number of periods in a time series.
`"simple"`
Factional change between first and last value. Maintains direction of growth by dividing the change
by the **absolute** value of the initial value:
$g = \frac{x_f - x_0}{|x_0|}$
`"average"`
The average fraction of change between consecutive values. Also maintains direction,
unlike e.g. pandas pct\_change function:
$g = \frac{1}{n} \sum_{i=1}^{n} \frac{x_i - x_{i-1}}{|x_{i-1}|}$
`"compound"`
Analogous to [CAGR](https://www.investopedia.com/terms/c/cagr.asp) (Compound Annual Growth Rate).
The average growth rate over the entire period, assuming the growth is compounded:
$g = \left( \frac{x_f}{x_0} \right)^{\frac{1}{n}} - 1$
`"linear"`
Fits a linear regression to the time series and returns the slope of the line.
Each item in array.
Values must be one of the following:
* `simple`
* `average`
* `compound`
* `linear`
* E.g. deriving two features from catch22 and growth sets each:
```json theme={null}
{
"catch22": ["mode_5", "acf_timescale"],
"growth": ["simple", "linear"],
}
```
Frequency to use by features in the TSFeatures set.
The number of observations in a single cycle. Used by certain features (for now only in the tsfeatures set),
that are based on seasonality. When a string (character) is provided, this is interpreted as the natural frequency
of the time series and will be translated to the number of observations per cycle using the following mapping:
* 'H': 24 (hourly)
* 'D': 1 (daily)
* 'M': 12 (monthly)
* 'Q': 4 (quarterly)
* 'W': 1 (weekly)
* 'Y': 1 (yearly)
E.g. if the natural frequency of the time series is monthly ('M'), will
analyze seasonality with a period of 12 observations (months in a year). If a number is provided,
this will be interpreted directly as the number of observations per cycle. If `null`, attempts to
infer the frequency automatically.
Also see [this post](https://robjhyndman.com/hyndsight/seasonal-periods/) by the author of
the original `tsfeatures` package for more details on seasonality and the frequency parameter.
Temporal unit to use.
Only required for converting the time column to timestamps when it is numeric.
Y=years, M=months, W=weeks, D=days, h=hours, m=minutes, s=seconds,
ms=milliseconds, us=microseconds, ns=nanoseconds.
Values must be one of the following:
`Y` `M` `W` `D` `h` `m` `s` `ms` `us` `ns`
Output format.
The format of the output dataset. The following options are supported:
* "wide": One row per time series with features as multivalues (list) columns
* "tall": Features joined to the original data, preserving all rows.
Values must be one of the following:
* `wide`
* `tall`
Number of parallel jobs.
If -1, all processors are used. If 1, no parallel computing code is used at all,
which is useful for debugging. Using multiple processes with a large dataset may
cause memory issues.
Values must be in the following range:
```javascript theme={null}
-1 ≤ n_jobs < inf
```
# group_by
Source: https://docs.graphext.com/api-docs/prepare/aggregate/group_by
Group data by specified columns and apply aggregation functions to each group.
## Usage
The following examples show how the step can be used in a recipe.
This example groups the dataset by an exact match on the `category` column and a date component (month level) on the `date` column, and then aggregates the count of `sales` and the sum of `revenue`:
```stan theme={null}
group_by(ds, {
"by": [
{ "by": "category", "groupingType": "EXACT" },
{ "by": "date", "groupingType": "DATE_COMPONENT", "param": { "component": "MONTH", "timezone": "UTC" } }
],
"aggregations": [
{ "name": "total_sales", "on": "sales", "type": "COUNT" },
{ "name": "total_revenue", "on": "revenue", "type": "SUM" }
]
}) -> (ds_grouped)
```
This example uses the simplified `by` parameter to group by an exact match on `category`. The aggregation calculates the average of `revenue` for each group:
```stan theme={null}
group_by(ds, {
"by": ["category"],
"aggregations": [
{ "name": "average_revenue", "on": "revenue", "type": "AVG" }
]
}) -> (ds_grouped)
```
This example groups by `category` and creates a sorted list of `values` based on the `sortCol` column:
```stan theme={null}
group_by(ds, {
"by": ["category"],
"aggregations": [
{ "name": "sorted_values", "on": "values", "type": "LIST", "params": { "value": "sortCol" } }
]
}) -> (ds_grouped)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
group_by(ds_in: dataset, {
"param": value,
...
}) -> (ds_out: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
The input dataset containing the columns to group by and apply aggregations on.
A dataset containing the aggregated results based on the grouping operations.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Columns to group by.
An array specifying the columns used for grouping. The `by` parameter can be either:
* An array of column names (e.g., `["column1", "column2"]`), which defaults to `EXACT` grouping.
* An array of objects with `by`, `groupingType`, optional `name` and optional `param` properties.
Column name to group by.
Column to group by.
Name of the output column. It is optional and defaults to the column name (`by` parameter).
Type of grouping operation.
The type of grouping operation. You can group by exact value match, a date component,
a range of numerical values, or quantiles.
Values must be one of the following:
* `EXACT`
* `DATE_COMPONENT`
* `RANGE`
* `QUANTILES`
Grouping parameters.
Date component to group by.
Date component to group by (only for `DATE_COMPONENT` grouping type).
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Timezone to use for date grouping.
The timezone to apply when grouping by date component.
Date interval to use.
The interval unit to apply when grouping by range.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `WEEK` `MONTH` `QUARTER` `YEAR`
Count of intervals in each range.
Number of intervals in each range when `groupingType` is `RANGE`.
Number of bins.
Number of bins to divide the data into.
Values must be in the following range:
```javascript theme={null}
1 ≤ nBins < inf
```
Use pretty bins.
Whether to adjust bin edges to be more human-readable.
Range size as number for `RANGE` grouping or number of quantiles.
Specify a range size directly as a number when `groupingType` is `RANGE`
or number of quantiles when `groupingType` is `QUANTILES`.
Null is accepted when no param is needed for the grouping type.
Aggregation functions to apply.
An array specifying the aggregation functions to apply on each group.
The array can be empty, in which case no aggregations are performed, but the dataset is still grouped by the specified columns.
Name of the output column.
Column on which the aggregation is applied.
If null, the aggregation applies to the entire group.
Type of aggregation function.
The type of aggregation function to perform on the specified column.
Includes support for standard aggregations (e.g., `SUM`, `COUNT`) as well as element-wise aggregations.
Notes:
* `PERCENT_OF_ROWS_WHERE`: Computes the percentage **within each group** where a condition is true.
* `PERCENT_OF_ROWS`: Computes the percentage **relative to the total number of rows** across all groups.
Values must be one of the following:
`COUNT` `MIN` `MAX` `SUM` `AVG` `VARIANCE` `STDEV` `FIRST` `LAST` `P25` `P50` `P75` `COUNT_WHERE` `NUMBER_OF_ROWS` `NUMBER_OF_ROWS_WHERE` `PERCENT_OF_ROWS` `PERCENT_OF_ROWS_WHERE` `METRIC` `MODE` `UNIQUE_VALUES` `LIST_UNIQUE` `LIST` `CONCATENATE` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV` `ELEMENT_FIRST` `ELEMENT_LAST`
Additional parameters for specific aggregations.
value.
Additional value used for certain aggregation types (e.g., `COUNT_WHERE`, `METRIC`, or `LIST` for presorting).
The *graphext advanced query* used to identify the rows to select previous to the grouping.
# Aggregate
Source: https://docs.graphext.com/api-docs/prepare/aggregate/index
| Step | Fast | Description |
| --------------------------------------------------------------------------------------- | ---- | ----------------------------------------------------------------------------------------------------------------------- |
| [aggregate](/api-docs/prepare/aggregate/aggregate) | | Group and aggregate a dataset using any of a number of predefined functions |
| [aggregate\_list\_items](/api-docs/prepare/aggregate/aggregate_list_items) | | Group a dataset by elements in a column of lists and aggregate remaining columns using one or more predefined functions |
| [aggregate\_neighbours](/api-docs/prepare/aggregate/aggregate_neighbours) | | For each node in a network, group and aggregate over its neighbours |
| [aggregate\_tweets\_by\_author](/api-docs/prepare/aggregate/aggregate_tweets_by_author) | | Group a dataset of tweets by author and calculate relevant author statistics |
| [featurize\_time\_series](/api-docs/prepare/aggregate/featurize_time_series) | | Summarizes time series data into aggregate metrics |
| [group\_by](/api-docs/prepare/aggregate/group_by) | ⚡ | Group data by specified columns and apply aggregation functions to each group |
| [melt](/api-docs/prepare/aggregate/melt) | ⚡ | Reshape a dataset by transforming columns into rows |
| [resample](/api-docs/prepare/aggregate/resample) | | Resamples a dataset of events or time series to the desired frequency |
# melt
Source: https://docs.graphext.com/api-docs/prepare/aggregate/melt
Reshape a dataset by transforming columns into rows.
This process involves transforming a dataset by first optionally organizing its rows based on certain criteria.
It then identifies unique values or combinations thereof within specified columns,
creating groups based on these unique identifiers. For each group,
specific functions are applied to the rows to summarize or condense their information.
The result is a new dataset where each row represents a unique group,
and each column corresponds to the outcome of a distinct summarization function applied across the grouped data.
For more information, refer to the [pandas melt documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.melt.html).
## Usage
The following example shows how the step can be used in a recipe.
Given a dataset `sales` with columns for `date`, `product_id`, and `sales_amount`,
we can use melt to transform this dataset into a long format.
This transformation will create a new dataset `long_sales`
where each row represents a single observation of sales amount for a product on a given date,
facilitating further analysis or visualization.
The resulting dataset `long_sales` will have the following columns:
* `date`: the date of the observation
* `variable`: indicating the product by its `product_id`
* `value`: the sales amount for that product on the given date
```stan theme={null}
melt(ds, {
"id_vars": ["date"],
"value_vars": ["product_id", "sales_amount"],
"var_name": "variable",
"value_name": "value"
}) -> (ds_melted)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
melt(ds_in: dataset, {
"param": value,
...
}) -> (melted: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Result of melting ds.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Identifier columns.
The column(s) to use as identifier variables.
Each item in array.
* order\_id
* \['customer\_id', 'order\_id']
Value columns.
The column(s) that are considered as value variables.
Each item in array.
* quantity
* \['price', 'quantity', 'discount']
Variable column name.
Name of the variable column if not provided, we will use the name 'variable'.
Value column name.
Name of the value column if not provided, we will use the name 'value'.
# resample
Source: https://docs.graphext.com/api-docs/prepare/aggregate/resample
Resamples a dataset of events or time series to the desired frequency.
Converts raw timestamped events, or contiguous time series data, from their original frequency
to daily, weekly, monthly, quarterly, yearly or other frequencies. Essentially, groups and aggregates
each time or event series by the specified time period and applies desired aggregations
(count of events, total spend etc.).
The step accepts input data, and can generate output data, in both a `tall` and a `wide` format:
* **Tall format**\
Each row represents a single event or observation, and the dataset contains *scalar* columns
for the event's timestamp as well as for identifying the series, customer or entity the event belongs to.
This is the most common format for event data and the most probable to have been imported in Graphext.
* **Wide format**\
Each row represents a single entity (customer), and the dataset contains columns of *lists* containing the event
timestamps and values for each series or observation. In this case, all lists in the same row must have
the same length. This format is the most convenient for analysis in Graphext, as it allows for easy
exploration of the time series data. You maintain one row per customer (entity), instead of duplicating
the customer's information for each event, yet you can still access, plot and generally work with all of the
customer's time series.
Note that both formats have the same number of columns. The difference is that in the "tall" format, each row
represents a single event, while in the "wide" format, each row represents a single entity and contains all its
events. You can think of the wide format as the result of aggregating by the time series identifier, and collecting
the timestamps and values in parallel columns of lists.
You can use the parameters below to configure the frequency to resample the data to, the format of the output
dataset (tall vs wide), whether to fill gaps in the resampled data etc.
## Usage
The following example shows how the step can be used in a recipe.
This examples resamples a dataset `ds` of shopping events in "tall" format to a weekly frequency, calculating the number of events per customer per week, weekly total and average spend, and the percentage of purchases in the category "pet food". The original value columns to be aggredated are "price" and "category". It also requests the output to be in "wide" format, which is the most convenient in Graphext. Setting `fill_gaps` to `true` ensures that the resampled data contains rows for all weeks between a customer's first and last event, even those with no events.
```stan theme={null}
resample(ds, {
"id": "customer_id",
"time": "timestamp",
"freq": "W",
"output": "wide",
"fill_gaps": true,
"aggregations": {
"price": {
"total_spend": {"func": "sum"},
"avg_spend": {"func": "mean"}
},
"id": {
"num_items_purchased": {"func": "count"}
},
"category": {
"pct_pet_food": {"func": "percent_where", "value": "pet food"}
}
}
}) -> (weekly)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
resample(ds_in: dataset, {
"param": value,
...
}) -> (ds_out: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A dataset of events or time series to resample. Must have id, timestamp and at least one value column.
The resampled dataset in the configured format. Will have id and timestamp columns, as well as one for
each aggregation function specified in the `aggregations` parameter. The timestamp and aggregated value
columns will have scalar values if the output was requested in "tall" format, or lists if in "wide" format.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Entity identifier(s).
I.e. name of the column(s) containing the entity identifier. If there are multiple time series
in your dataset, e.g. one per customer, this is the column that identifies the series. If there are
multiple ID columns, the time series will be grouped by the unique combination of values in these columns.
Each item in array.
* customer\_id
* \['last\_name', 'first\_name', 'birthday']
Timestamp.
Name of the column containing the event timestamp.
Frequency.
Alias of the frequency to resample the data to. The following are the possible values for the `freq` parameter.
Also see the corresponding [Pandas documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#dateoffset-objects)
for more details on each frequency.
| Alias | Description | |
| :---------- | :-------------------------------------------------------- | - |
| B | Business day (weekday) | |
| D | Calendar day (absolute) | |
| W | Week, optionally anchored on a day of the week (W-SUN...) | |
| ME | Calendar month end (last day of month) | |
| SME | Semi-month end (15th and end of month) | |
| BME | Last business day of month | |
| MS | Calendar month start (first day mof month) | |
| SMS | Semi-month start (1st and 15th) | |
| BMS | First business day of month | |
| QE | Calendar quarter end | |
| BQE | Business quarter end | |
| QS | Calendar Quarter start | |
| BQS | Business quarter start | |
| YE/A/Y | Calendar year end | |
| BYE/BA/BY | Business year end | |
| YS/AS/YS | Calendar year start | |
| BYS/BAS/BYS | Business year start | |
| h/H | Hour | |
| bh/BH | Business hour | |
| min/T | Minute | |
| s/S | Second | |
| ms/L | Millisecond | |
| us/U | Microsecond | |
| ns/N | Nanosecond | . |
Values must be one of the following:
`B` `D` `W` `M` `ME` `SM` `SME` `BM` `BME` `MS` `SMS` `BMS` `Q` `QE` `BQ` `BQE` `QS` `BQS` `A` `Y` `YE` `BA` `BY` `BYE` `AS` `BAS` `BYS` `h` `H` `bh` `BH` `T` `min` `S` `s` `L` `ms` `U` `us` `N` `ns`
Anchor.
The date to anchor the resampling on. For example, if the frequency is "W" (weekly) and the anchor is "WED",
the resampling will be done on periods between consecutive Wednesdays. If the frequency is "YS" (yearly) and
the anchor is "JUL", the resampling will be annually with each period ending at the end of July. The anchor
can only be used with weekly, quarterly and yearly frequencies.
Values must be one of the following:
`MON` `TUE` `WED` `THU` `FRI` `SAT` `SUN` `JAN` `FEB` `MAR` `APR` `MAY` `JUN` `JUL` `AUG` `SEP` `OCT` `NOV` `DEC`
Output format.
The format of the output dataset. In "tall" format, each row represents a single event or observation,
and columns contain scalar values. In the "wide" format, each row represents a single entity, and the
dataset contains columns of lists (of the same length within each row).
Values must be one of the following:
* `tall`
* `wide`
Fill gaps.
Whether to fill gaps in the resampled data with `NaN`/0 values. If set to `false`, the resampled data will
only contain rows for which there are events in the original data. If set to `true`, the resampled data will
contain rows for all periods in the resampled frequency, with `NaN`/0 values for periods with no events.
Definition of desired aggregations.
A dictionary mapping original columns to new aggregated columns, specifying an aggregation function for each.
*Aggregations* are functions that reduce all the values in a particular column of a single group to a single summary value of that group.
E.g. a `sum` aggregation of column A calculates a single total by adding up all the values in A belonging to each group.
In contrast to the more generic `aggregate` and `group_by` steps, for time series resampling, only functions returning scalar values
are supported. Allowed options for the `func` parameters are:
* `n`, `size` or `count`: calculate number of rows in group
* `sum`: sum total of values
* `mean`: take mean of values
* `max`: take max of values
* `min`: take min of values
* `first`: take first item found
* `last`: take last item found
* `n_unique`: count the number of unique values
* `concatenate`: convert all values to text and concatenate them into one long text
* `count_where`: number of rows in which the column matches a value, needs parameter `value` with the value that you want to count
* `percent_where`: percentage of the column where the column matches a value, needs parameter `value` with the value that you want to count
Note that in the case of `count_where` and `percent_where` an additional `value` parameter is required.
One item per input column.
Each key should be the name of an input column, and each value an object defining one or more aggregations for that column.
An individual aggregation consists of the name of a desired output column, mapped to a specific aggregation function.
For example:
```json theme={null}
{
"input_col": {
"output_col": {"func": "sum"}
}
}
```
Object defining how to aggregate a single output column.
Needs at least the `"func"` parameter. If the aggregation function accepts further arguments,
like the `"value"` parameter in case of `count_where` and `percent_where`, these need to be provided also.
For example:
```json theme={null}
{
"output_col": {"func": "count_where", "value": 2}
}
```
Aggregation function.
Values must be one of the following:
`n` `size` `count` `sum` `mean` `n_unique` `count_where` `percent_where` `concatenate` `max` `min` `first` `last` `list`
* Including an aggregation function with additional parameters:
```json theme={null}
{
"product_id": {
"products": {"func": "list"},
"size": {"func": "count"}
},
"item_total": {
"total": {"func": "sum"},
},
"item_category": {
"num_food_items": {"func": "count_where", "value": "food"}
}
}
```
# caption_images
Source: https://docs.graphext.com/api-docs/prepare/enrich/caption_images
Predict image captions using pretrained DL models.
In its current form the step predicts image captions using [ClipClap](https://github.com/rmokady/CLIP_prefix_caption).
ClipClap first embeds images using the [Clip](https://huggingface.co/docs/transformers/model_doc/clip) model,
which has been pre-trained on 400M image/text pairs to pick out an image's correct caption from a list of candidates. These
images are then projected into the embedding space of the [GPT-2](https://huggingface.co/gpt2) language model, using a
custom model trained for the task. Finally, using this projection as a prefix, the pretained GPT-2 is asked to predict the
next sentence, i.e. the one following the image.
## Usage
The following example shows how the step can be used in a recipe.
The step has no required parameters, so the simplest call is simply
```stan theme={null}
caption_images(ds.image_url) -> (ds.caption)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
caption_images(images: url, {
"param": value,
...
}) -> (caption: text)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column of URLs to images to predict captions for.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Which projection model to use.
The projection model maps embeddings from the pretrained Clip image model, to the pretrained
GPT-2 language model. Select between a multi-layer perceptron ("MLP"), or the faster transformer
("TRF").
Values must be one of the following:
* `TRF`
* `MLP`
Select the parameter set for the model.
The ClipClap authors provide weights for models having been trained either on the
[COCO dataset](https://cocodataset.org/#home) ("coco") or the [ConceptualCaptions](https://ai.google.com/research/ConceptualCaptions/)
dataset ("concept").
Values must be one of the following:
* `coco`
* `concept`
Whether to use beam-search or greedy word prediction.
When enabled, uses a more expensive but "smarter" algorithm to predict the words in the captions.
# classify_text
Source: https://docs.graphext.com/api-docs/prepare/enrich/classify_text
Classify texts using any model from the [Hugging Face hub](https://huggingface.co/models).
Note that we do not validate the model name before executing it, so make sure it
corresponds to an existing model in the hub, otherwise the step will fail.
## Usage
The following example shows how the step can be used in a recipe.
To infer the ternary sentiment of tweets using a CardiffNLP model
```stan theme={null}
classify_text(ds.text, {"model": "cardiffnlp/twitter-roberta-base-sentiment"}) -> (ds.text_sentiment)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
classify_text(text: text, {
"param": value,
...
}) -> (class: category)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column of texts to classify.
The inferred class of each text. The labels of individual categories depend on the seleted model,
and/or can be specified manually using the `labels` parameter (see below).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
The name of a model.
This should be the full name (including the organization if applicable) of a model in the
[Hugging Face model hub](https://huggingface.co/models). You can copy it by clicking on the
icon next to the model's name on its dedicated web page.
Note that if the name doesn't correspond to a model existing in the hub the step will fail.
Since there are hundreds if not thousands of potential models, we cannot validate if the
name is correct before executing it.
The specific model version.
Can be a branch name, a tag name, or a commit id. To identify a particular revision, on
a model's webpage (such as [https://huggingface.co/cardiffnlp/twitter-xlm-roberta-base-sentiment-multilingual](https://huggingface.co/cardiffnlp/twitter-xlm-roberta-base-sentiment-multilingual)),
browse to the [Files and versions tab](https://huggingface.co/cardiffnlp/twitter-xlm-roberta-base-sentiment-multilingual/tree/main),
and use the branch or history dropdown menus to see the available branch names or commit IDs.
If not provided, will use the latest (newest) available version (usually from the "main" branch).
Map original model output to human-readable labels.
Unfortunately, many models in Hugging Face are badly configured and output labels like `LABEL_0`,
`LABEL_1`, etc. which isn't very helpful. You can use the "Hosted inference API"
widget on the model's web page to test its output labels. If necessary, use this parameter
to map the default output labels to ones you prefer.
One or more additional parameters.
* E.g. to map ternary sentiment labels
```json theme={null}
"labels": {
"LABEL_0": "negative",
"LABEL_1": "neutral",
"LABEL_2": "positive"
}
```
Minimum probability (score) to accept prediction label.
Class labels with a corresponding probability smaller than this value will be removed
(replaced with NaN, i.e. the missing value).
Values must be in the following range:
```javascript theme={null}
0.0 < min_prob < 1.0
```
How many texts to process simultaneously.
May get ignored when running on CPU.
Values must be in the following range:
```javascript theme={null}
1 ≤ batch_size ≤ 64
```
Number of threads used to feed GPU with texts.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_workers ≤ 4
```
Which CPU/GPU to run model on.
Pass -1 to use CPU, and 0 to use first available GPU. By default, of
when passed `null`, the step will use GPU automatically if one is found
otherwise CPU.
ID of a Hugging Face integration configured in Graphext.
To use a private model from the Hugging Face hub, you need to configure a
Hugging Face "API Key" integration (in the relevant Graphext team > Add Integration
> API KEYS > Add API Key > Hugging Face > paste an access token previously
> configured in your huggingface account). Graphext will automatically assign
> an ID to your integration which gets autocompleted where required (e.g. in the
> recipe editor).
# clean_categories
Source: https://docs.graphext.com/api-docs/prepare/enrich/clean_categories
Clean a given column of categories or lists of categories using OpenAI.
## Usage
The following example shows how the step can be used in a recipe.
Bring the number of categories down to 10 categories
```stan theme={null}
clean_categories(ds.categories, {
"integration": "open-ai-1",
"instructions": "Generate around 10 categories.",
"model": {
"id": "gpt-4.1-mini",
"temperature": 0.7
}
}) ->(ds.cleaned_categories)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
clean_categories(original: category|list[category], {
"param": value,
...
}) -> (cleaned: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
original column.
cleaned column.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Associated integration.
Categories desired in the result column. If passed instructions are ignored.
Each item in array.
Further instructions to generate the desired set of categories. You can ask for things like 'generate around 5 categories'
Approximate number of categories to generate.
Can be a float from 0 to 1 (percentage of unique categories) or an integer greater than 1.
number.
Values must be in the following range:
```javascript theme={null}
0 ≤ {_} ≤ 1
```
integer.
Values must be in the following range:
```javascript theme={null}
2 ≤ {_} < inf
```
Model Configuration.
Configuration for OpenAI's model.
OpenAI model to choose.
Values must be one of the following:
`gpt-4.1` `gpt-4.1-mini` `gpt-4.1-nano` `gpt-5-mini` `gpt-5-nano` `o4-mini`
Temperature. Higher means more creativity, but also makes the model more likely to hallucinate. Lower temperature yields more deterministic results. Ignored for reasoning models (gpt-5-mini, gpt-5-nano, o4-mini).
Values must be in the following range:
```javascript theme={null}
0 ≤ temperature ≤ 1
```
# describe_clusters
Source: https://docs.graphext.com/api-docs/prepare/enrich/describe_clusters
Describe your clusters from their given relevant TF-IDF terms using OpenAI.
## Usage
The following example shows how the step can be used in a recipe.
Get description from tf-idf cluster keywords
```stan theme={null}
describe_clusters(ds.tfidf_clusters) -> (ds.description)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
describe_clusters(tfidf: category, {
"param": value,
...
}) -> (descriptions: category)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
TF-IDF keywords from the clusters to describe in natural language.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Language.
Language you wish to use. Defaults to the TF-IDF keywords.
Model Configuration.
Configuration for OpenAI's model.
OpenAI model to choose.
Values must be one of the following:
`gpt-4.1` `gpt-4.1-mini` `gpt-4.1-nano` `gpt-5-mini` `gpt-5-nano` `o4-mini`
Temperature. Higher means more creativity, but also makes the model more likely to hallucinate. Lower temperature yields more deterministic results. Ignored for reasoning models (gpt-5-mini, gpt-5-nano, o4-mini).
Values must be in the following range:
```javascript theme={null}
0 ≤ temperature ≤ 1
```
# explore_database
Source: https://docs.graphext.com/api-docs/prepare/enrich/explore_database
Explore database structure — generates a dataset of column metadata with statistics, relationships, and semantic descriptions.
## Usage
The following shows how the step can be used in a recipe.
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
explore_database(, {
"param": value,
...
}) -> (ds_out: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Database Integration.
ID of your database integration containing the connection credentials.
Schemas to explore.
Comma-separated list of schemas to inspect. If empty, all non-system schemas are explored automatically.
Skip per-column queries.
Skip sample values and min/max/avg/stddev (faster for large databases).
Generate graph link columns.
Generate targets and weights columns encoding the database structure as a graph.
Links include column-to-table, table-to-schema, and FK relationships (directed).
Weight for column-to-table links.
Weight assigned to directed links from column nodes to their table hub node. Set to 0 to disable.
Values must be in the following range:
```javascript theme={null}
0 ≤ table_link_weight < inf
```
Weight for table-to-schema links.
Weight assigned to directed links from table hub nodes to their schema hub node. Set to 0 to disable.
Values must be in the following range:
```javascript theme={null}
0 ≤ schema_link_weight < inf
```
Weight for FK links.
Weight assigned to directed links from FK columns to their target columns. Set to 0 to disable.
Values must be in the following range:
```javascript theme={null}
0 ≤ fk_link_weight < inf
```
# fetch_demographics_es
Source: https://docs.graphext.com/api-docs/prepare/enrich/fetch_demographics_es
Fetch Spanish demographic census data given a geographical location in each row.
Using geographical coordinates provided by latitude and longitude columns, enriches the input dataset with
a subset of the 2011 Spanish census. See below for the seven features to be added by the step.
## Usage
The following example shows how the step can be used in a recipe.
Since the step has no configuration parameters, it's simply
```stan theme={null}
fetch_demographics_es(ds.lat, ds.lon) -> (
ds.Dem_Extranjeros_pct,
ds.Dem_Casados_pct,
ds.Dem_Edad,
ds.Dem_Nivel_Estudios,
ds.Dem_Vivienda_m2,
ds.Dem_Personas_Hogar,
ds.Dem_Location_Idx
)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
fetch_demographics_es(lat: number, lon: number
) -> (
Dem_Extranjeros_pct: number,
Dem_Casados_pct: number,
Dem_Edad: number,
Dem_Nivel_Estudios: number,
Dem_Vivienda_m2: number,
Dem_Personas_Hogar: number,
Dem_Location_Idx: number
)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A numeric column containing latitude coordinates of the places of interest.
A numeric column containing longitude coordinates of the places of interest.
A numeric column containing the percentage of immigrants registered in the area.
A numeric column containing the percentage of married people in the area.
A numeric column containing the average age in the area.
A numeric column containing the average level of education.
A numeric column containing the average size of households in square meters.
A numeric column containing the average number of household members.
A numeric column containing an index in the range 1-9 summarizing the overall quality of the location; lower
values being better.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
This step doesn't expect any configuration.
# fetch_full_contact_domains
Source: https://docs.graphext.com/api-docs/prepare/enrich/fetch_full_contact_domains
Enrich a dataset containing links (URLs) to companies' online presence using the FullContact service.
Will produce information about each company's social media accounts (including twitter, facebook, linkedin, etc.) and
other general information (industries it belongs to, location of its headquarter etc.). See below for the full list
of enrichment columns added by the step.
???+ info "API integration"
To use this step your team needs to have the *Full Contact* integration configured in Graphext. The corresponding credentials
are required to connect to a third-party API. You can configure API integrations following the `INTEGRATIONS` or `ADD INTEGRATION`
link in the top-left corner of your Team's page, selecting `API keys`, and then the name of the desired third-party service.
To enable the *Full Contact* integration in particular you will need access to Full Contact's service. Follow the instructions
[here](https://www.fullcontact.com/developer-portal/) to create the required API key.
## Usage
The following example shows how the step can be used in a recipe.
Since the step has no configuration parameters, it's simply
```stan theme={null}
fetch_full_contact_domains(ds.company) -> (
ds.domain,
ds.name,
ds.location,
ds.twitter,
ds.linkedin,
ds.facebook,
ds.bio,
ds.logo,
ds.website,
ds.founded,
ds.employees,
ds.locale,
ds.category,
ds.angellist_profile,
ds.angellist_bio,
ds.angellist_followers,
ds.twitter_url,
ds.twitter_bio,
ds.twitter_followers,
ds.twitter_following,
ds.crunchbase_url,
ds.crunchbase_bio,
ds.linkedin_url,
ds.linkedin_bio,
ds.linkedin_followers,
ds.facebook_url,
ds.location_formatted,
ds.industries,
ds.keywords,
ds.global_traffic_rank
)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
fetch_full_contact_domains(domain: category, {
"param": value,
...
}) -> (
name: category,
location: category,
twitter: category,
linkedin: category,
facebook: category,
bio: text,
logo: url,
website: url,
founded: number,
employees: number,
locale: category,
category: category,
angellist_profile: url,
angellist_bio: text,
angellist_followers: number,
twitter_url: url,
twitter_bio: text,
twitter_followers: number,
twitter_following: number,
crunchbase_url: url,
crunchbase_bio: text,
linkedin_url: url,
linkedin_bio: text,
linkedin_followers: number,
facebook_url: url,
location_formatted: category,
industries: list[category],
keywords: list[category],
global_traffic_rank: number
)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Categorical column containing domain of the company to fetch information for.
Categorical column containing the name of the company.
Last known location of the company.
Column containing the Twitter profile of the company.
Column containing the LinkedIn profile of the company.
Column containing the Facebook profile of the company.
Column containing the biography most relevant to the company from their social media profiles.
URL of the logo of the company.
URL pointing to the website of the company.
Date the company was founnded.
Approximation of the number of employees.
Locale relevant to the company.
Category the company belongs to.
URL of the AngelList company profile.
Biography from the AngelList company profile.
Number of followers the company currently has on AngelList.
URL of the Twitter company profile.
Biography from the Twitter company profile.
Number of followers the company currently has on Twitter.
Number of profiles the company currently follows on Twitter.
URL of the Crunchbase company profile.
Biography from the CrunchBase company profile.
URL of the LinkedIn company profile.
Biography from the LinkedIn company profile.
Number of profiles the company currently follows on LinkedIn.
URL of the Facebook company profile.
Last location known to the company's headquarters.
List of industries the company belongs to.
List of keywords the company is associated to.
Rank in the global web traffic for the company.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
ID of the integration you'd like to use.
# fetch_full_contact_emails
Source: https://docs.graphext.com/api-docs/prepare/enrich/fetch_full_contact_emails
Enrich a dataset containing email addresses with personal information using the _FullContact_ service.
???+ info "API integration"
To use this step your team needs to have the *Full Contact* integration configured in Graphext. The corresponding credentials
are required to connect to a third-party API. You can configure API integrations following the `INTEGRATIONS` or `ADD INTEGRATION`
link in the top-left corner of your Team's page, selecting `API keys`, and then the name of the desired third-party service.
To enable the *Full Contact* integration in particular you will need access to Full Contact's service. Follow the instructions
[here](https://www.fullcontact.com/developer-portal/) to create the required API key.
## Usage
The following example shows how the step can be used in a recipe.
The step has no configuration parameter, so it's simply
```stan theme={null}
fetch_full_contact_emails(ds.email) -> (
ds.fullName,
ds.age,
ds.gender,
ds.location,
ds.title,
ds.organization,
ds.twitter,
ds.linkedin,
ds.facebook,
ds.bio,
ds.avatar,
ds.website,
ds.given_name,
ds.family_name,
ds.education_institution,
ds.education_degree
)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
fetch_full_contact_emails(email: category, {
"param": value,
...
}) -> (
fullName: category,
age: number,
gender: category,
location: category,
title: category,
organization: category,
twitter: category,
linkedin: category,
facebook: category,
bio: text,
avatar: url,
website: url,
given_name: category,
family_name: category,
education_institution: category,
education_degree: category
)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Emails to fetch personal information for.
Full name of the person.
Age of the person.
Gender of the person (M/F).
Currently known location.
Current title the person holds at the organization.
Current organization a person belongs to.
Twitter profile.
LinkedIn profile.
Facebook profile.
Biography as extracted from social media accounts.
Picture uses in social media accounts.
Personal website.
Person's given name.
Person's family name.
Category containing the most relevant educational institution the person attended.
Category representing the degree obtained at this institution.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
ID of the integration you'd like to use.
# fetch_google_places
Source: https://docs.graphext.com/api-docs/prepare/enrich/fetch_google_places
Fetch information about the most relevant places surrounding a location.
Given the latitude and longitude coordinates of a location in each row, uses the *Google Places* API to enrich the
dataset with information about the surrounding businesses, their ratings etc. See below for further information about
the specific columns to be added by the step.
???+ info "API integration"
To use this step your team needs to have the *Google Places* integration configured in Graphext. The corresponding credentials
are required to connect to a third-party API. You can configure API integrations following the `INTEGRATIONS` or `ADD INTEGRATION`
link in the top-left corner of your Team's page, selecting `API keys`, and then the name of the desired third-party service.
To enable the *Google Places* integration in particular you will need access to Google's Places API. Follow the instructions
[here](https://developers.google.com/places/web-service/get-api-key) to create the required API key.
## Usage
The following example shows how the step can be used in a recipe.
To search places within a default radius of 1.5km and of arbitrary type
```stan theme={null}
fetch_google_places(ds.lat, ds.lon) -> (
ds.places_total,
ds.places_number_ratings,
ds.places_ratings_mean,
ds.places_number_types,
ds.places_types,
ds.places_names
)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
fetch_google_places(lat: number, lon: number, {
"param": value,
...
}) -> (
places_total: number,
places_number_ratings: number,
places_ratings_mean: number,
places_number_types: number,
places_types: list[category],
places_names: list[category]
)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Column containing the latitude of each location.
Column containing the longitude of each location.
Number of places the API returned, maximum 10.
Number of ratings on average for the places found, proxy for place popularity.
Average of the ratings of the places found, proxy for location index.
Number of different types of places found.
Types of the places found, for possible categories see [here](https://developers.google.com/places/web-service/supported_types).
Names of the places found.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
ID of the integration you'd like to use.
Search radius in meters around the given location.
Values must be in the following range:
```javascript theme={null}
10 ≤ radius < inf
```
Search for a specific type of place only.
For supported types also see the [Google Places documentation here](https://developers.google.com/places/web-service/supported_types).
Values must be one of the following:
`accounting` `airport` `amusement_park` `aquarium` `art_gallery` `atm` `bakery` `bank` `bar` `beauty_salon` `bicycle_store` `book_store` `bowling_alley` `bus_station` `cafe` `campground` `car_dealer` `car_rental` `car_repair` `car_wash` `casino` `cemetery` `church` `city_hall` `clothing_store` `convenience_store` `courthouse` `dentist` `department_store` `doctor` `drugstore` `electrician` `electronics_store` `embassy` `fire_station` `florist` `funeral_home` `furniture_store` `gas_station` `grocery_or_supermarket` `gym` `hair_care` `hardware_store` `hindu_temple` `home_goods_store` `hospital` `insurance_agency` `jewelry_store` `laundry` `lawyer` `library` `light_rail_station` `liquor_store` `local_government_office` `locksmith` `lodging` `meal_delivery` `meal_takeaway` `mosque` `movie_rental` `movie_theater` `moving_company` `museum` `night_club` `painter` `park` `parking` `pet_store` `pharmacy` `physiotherapist` `plumber` `police` `post_office` `primary_school` `real_estate_agency` `restaurant` `roofing_contractor` `rv_park` `school` `secondary_school` `shoe_store` `shopping_mall` `spa` `stadium` `storage` `store` `subway_station` `supermarket` `synagogue` `taxi_stand` `tourist_attraction` `train_station` `transit_station` `travel_agency` `university` `veterinary_care` `zoo`
Specific keyword to use in the search for places.
* burger
# fetch_google_vision
Source: https://docs.graphext.com/api-docs/prepare/enrich/fetch_google_vision
Analyze images given their URL using the Google Vision API.
Labels and categorizes images to indicate whether they contain violent, parodic, adult, medical or racy elements.
???+ info "API integration"
To use this step your team needs to have the *Google Vision* integration configured in Graphext. The corresponding credentials
are required to connect to a third-party API. You can configure API integrations following the `INTEGRATIONS` or `ADD INTEGRATION`
link in the top-left corner of your Team's page, selecting `API keys`, and then the name of the desired third-party service.
To enable the *Google Vision* integration in particular you will need access to Google's Vision service. Follow the instructions
[shere](https://cloud.google.com/vision/docs/setup) to create the required API key.
## Usage
The following example shows how the step can be used in a recipe.
The step has no configuration parameters, so it's simply
```stan theme={null}
fetch_google_vision(ds.image_url) -> (
ds.image_labels,
ds.image_contains_violence,
ds.image_is_spoof,
ds.image_has_adult_content,
ds.image_has_medical_content,
ds.image_is_racy
)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
fetch_google_vision(target_col: url, {
"param": value,
...
}) -> (
labels: list[category],
violence: category,
spoof: category,
adult: category,
medical: category,
racy: category
)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Direct URLs to the images to analyze, e.g. [https://upload.wikimedia.org/wikipedia/commons/c/c7/Madrid\_-\_El\_Oso\_y\_el\_Madro%C3%B1o.jpg](https://upload.wikimedia.org/wikipedia/commons/c/c7/Madrid_-_El_Oso_y_el_Madro%C3%B1o.jpg).
Lists of entities identified in the picture.
Indicates whether the image contains violence.
Indicates whether the image is a spoof (parody).
Indicates whether the image features adult content.
Indicates whether the image features medical content.
Indicates whether the image features racy content.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
ID of the integration you'd like to use.
# fetch_location
Source: https://docs.graphext.com/api-docs/prepare/enrich/fetch_location
Extract formatted address, locality, area, state, country and geographical coordinates from one or more address columns.
It is possible to specify a postal code as a standalone column, but this will be interpreted as American
without a proper prefix. To use Spanish postal codes indicate this using the "CP" prefix, e.g. "CP 28001".
To use this step your team needs to have the *Google Location* integration configured in Graphext. The corresponding credentials
are required to connect to a third-party API. You can configure API integrations following the `INTEGRATIONS` or `ADD INTEGRATION`
link in the top-left corner of your Team's page, selecting `API keys`, and then the name of the desired third-party service.
To enable the *Google Location* integration in particular you will need access to Google's geocoding service. Follow the instructions
[here](https://developers.google.com/maps/documentation/geocoding/get-api-key) to create the required API key.
## Usage
The following example shows how the step can be used in a recipe.
Since the step has no configuration parameters, simply use
```stan theme={null}
fetch_location(ds.address) -> (
ds.canonical_address,
ds.locality,
ds.area,
ds.state,
ds.country,
ds.lat,
ds.lon)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
fetch_location(*address_columns: text|category|number, {
"param": value,
...
}) -> (
formatted_address: category,
locality: category,
area: category,
state: category,
country: category,
lat: number,
lon: number
)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
One or more text or categorical columns containing all or parts of an address (e.g. street, city, postal code etc.).
If several parts are specified in a single column they should be separated by a comma.
Categorical column containing the original address in a standardized format.
Categorical column containing each address's locality.
Categorical column containing each address's area.
Categorical column containing each address's state or region.
Categorical column containing each address's country.
Numeric column containing the latitude coordinate of the address.
Numeric column containing the longitude coordinate of the address.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
ID of the integration you'd like to use.
# fetch_openreview
Source: https://docs.graphext.com/api-docs/prepare/enrich/fetch_openreview
Fetch publications submitted to one or more conferences via [OpenReview](https://openreview.net/).
## Usage
The following example shows how the step can be used in a recipe.
To fetch papers from ICML and NeurIPS conferences and workshops, the input column may contain the values
`["NeurIPS.*Conference", "neurips.*workship", "ICML.*Conference", "icml.*workshop"]`
To fetch the corresponding papers (the default parameters are optional):
```stan theme={null}
fetch_openreview(ds.invitations, {"search": true, "regex": true}) -> (papers)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
fetch_openreview(venues: url|category|text, {
"param": value,
...
}) -> (papers: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A new dataset containing details about the submitted publications.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Whether the inputs are venues to search for.
If `false`, expects explicit reference IDs to conference invitations on openreview\.net,
e.g. "ICLR.cc/2023/Conference/-/Blind\_Submission".
Whether the input values are regex patterns.
If `true`, the input values are treated as regex patterns to search for in the venue names.
If `false`, searches for exact substrings in venue names.
# fetch_social_shares
Source: https://docs.graphext.com/api-docs/prepare/enrich/fetch_social_shares
Fetch the number of times a Url was shared on Facebook.
## Usage
The following example shows how the step can be used in a recipe.
This step doesn't have any configuration parameters. Hence simply:
```stan theme={null}
fetch_social_shares(ds.url) -> (ds.num_social_shares)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
fetch_social_shares(urls: url) -> (shares: number)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column containing links (URLs) pointing to articles.
A numerical column containing the number of times each article was shared on Facebook.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
This step doesn't expect any configuration.
# fetch_twitter
Source: https://docs.graphext.com/api-docs/prepare/enrich/fetch_twitter
Enriches a dataset containing tweets with information about their authors.
In addition to author information also adds the location the tweet was send from (if available),
and indicates whether a tweet was retweeted or favourited at least once. See below for a
detailed list of the enrichment columns.
???+ info "API integration"
To use this step you will need to create an app associated to your Twitter account,
and obtain four items: the consumer\_key, the consumer\_secret, the token and the token\_secret.
To do so you can follow the steps [here](https://help.graphext.com/en/articles/3973608-how-to-obtain-twitter-api-keys-for-twitter-enrichment),
and then add them to a Graphext integration.
## Usage
The following example shows how the step can be used in a recipe.
This step doesn't have any configuration parameters. Hence simply:
```stan theme={null}
fetch_twitter(ds.id) -> (
ds.user_created_at,
ds.user_description,
ds.user_favourites_count,
ds.user_followers_count,
ds.user_following_count,
ds.user_listed_count,
ds.user_profile_image_url_https,
ds.user_protected,
ds.user_tweets_count,
ds.user_profile_url,
ds.user_verified,
ds.user_location,
ds.lat,
ds.long,
ds.retweeted,
ds.favorited,
ds.source
)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
fetch_twitter(tweet_ids: category|number, {
"param": value,
...
}) -> (
user_created_at: date,
user_description: text,
user_favourites_count: number,
user_followers_count: number,
user_following_count: number,
user_listed_count: number,
user_profile_image_url_https: url,
user_protected: boolean,
user_tweets_count: number,
user_profile_url: url,
user_verified: boolean,
user_location: category,
lat: number,
long: number,
retweeted: boolean,
favorited: boolean,
source: category
)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Twitter's unique identifier for each tweet.
Time the user who tweeted this tweet was created.
Self-description of the user.
How many tweets the user has favorited.
How many followers the user has.
How many users the user is following.
How many times the user has been listed in public listings.
User's picture.
Whether this user has protected their tweets or not.
How many times this user has tweeted.
User's profile url.
Whether this user is verified or not.
Location extracted from the author profile.
Latitude this tweet was tweeted from.
Longitude this tweet was tweeted from.
Whether you have retweeted this tweet.
Whether you have favorited this tweet.
Where this tweet was tweeted from as a HTML string.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
ID of the integration you'd like to use.
# fetch_twitter_api_io
Source: https://docs.graphext.com/api-docs/prepare/enrich/fetch_twitter_api_io
Fetch tweets from Twitter/X using TwitterAPI.io advanced search.
Fetches tweets from Twitter/X using the TwitterAPI.io advanced search API.
Uses Twitter's native advanced search syntax (from:, to:, lang:, since:, until:, OR, etc.).
Returns a dataset with 37 columns: tweet text, author info, engagement metrics,
entities (hashtags, URLs, mentions), media URLs, and quoted/retweeted tweets.
Cost: \~\$0.15 per 1,000 tweets (15 credits per tweet).
## Query Syntax
| Operator | Example | Description |
| ------------------ | ------------------ | ------------------------- |
| `from:` | `from:NASA` | Tweets from a user |
| `to:` | `to:NASA` | Tweets replying to a user |
| `lang:` | `lang:es` | Filter by language |
| `since:` | `since:2026-01-01` | Tweets after date |
| `until:` | `until:2026-02-01` | Tweets before date |
| `OR` | `cat OR dog` | Either term |
| `-` | `cat -dog` | Exclude term |
| `""` | `"exact phrase"` | Exact match |
| `min_faves:` | `min_faves:100` | Minimum likes |
| `filter:media` | `filter:media` | Only with media |
| `-filter:retweets` | | Exclude retweets |
## Error Handling
* **Rate limit (429):** Retries with exponential backoff
* **Credits exhausted (402):** Returns partial data with warning (W41)
* **Timeout:** Returns partial data with warning (W41)
Learn more at [docs.twitterapi.io](https://docs.twitterapi.io).
## Usage
The following example shows how the step can be used in a recipe.
Fetch tweets about climate change in English
```stan theme={null}
fetch_twitter_api_io({
"integration": "MY_TWITTER_API_IO",
"query": "\"climate change\" lang:en since:2026-01-01",
"query_type": "Latest",
"max_items": 5000
}) -> (ds_out)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
fetch_twitter_api_io(, {
"param": value,
...
}) -> (ds_out: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
The dataset containing tweets from the search.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
TwitterAPI.io Integration.
ID of your TwitterAPI.io integration containing the API key.
Search Query.
Twitter advanced search query. Supports operators: from:, to:, lang:,
since:YYYY-MM-DD, until:YYYY-MM-DD, OR, -, "exact phrase",
min\_replies:, min\_faves:, filter:links, filter:media.
Reference: [https://github.com/igorbrigadir/twitter-advanced-search](https://github.com/igorbrigadir/twitter-advanced-search).
Sort Order.
Sort order. "Latest" for chronological, "Top" for most relevant.
Values must be one of the following:
* `Latest`
* `Top`
Max Tweets.
Maximum number of tweets to fetch. Leave empty to fetch all matching tweets.
Cost: \~15 credits per tweet (\$0.15 per 1,000 tweets).
Timeout.
Maximum seconds to wait for the search to complete.
# fetch_url_content
Source: https://docs.graphext.com/api-docs/prepare/enrich/fetch_url_content
Fetch the main text from a web URL, and return its title, author, content, excerpt and domain.
## Usage
The following example shows how the step can be used in a recipe.
Since the step has no configuration parameter, it's simply
```stan theme={null}
fetch_url_content(ds.article_url) -> (
ds.article_title,
ds.article_author,
ds.article_content,
ds.article_excerpt,
ds.article_domain
)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
fetch_url_content(urls: url) -> (
title: text,
author: category,
content: text,
excerpt: text,
domain: url
)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column of URLs linkling to articles, blog posts or webpages.
A text column containing the extracted article's title.
A categorical column containing the extracted article's author.
A text column containing the extracted article's main text.
A text column containing a summary of the extracted article.
A column containing only the domain of each original URL.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
This step doesn't expect any configuration.
# Enrich
Source: https://docs.graphext.com/api-docs/prepare/enrich/index
| Step | Fast | Description |
| ------------------------------------------------------------------------------------ | ---- | ----------------------------------------------------------------------------------------------------------------------- |
| [caption\_images](/api-docs/prepare/enrich/caption_images) | | Predict image captions using pretrained DL models |
| [classify\_text](/api-docs/prepare/enrich/classify_text) | | Classify texts using any model from the [Hugging Face hub](https://huggingface.co/models) |
| [clean\_categories](/api-docs/prepare/enrich/clean_categories) | | Clean a given column of categories or lists of categories using OpenAI |
| [describe\_clusters](/api-docs/prepare/enrich/describe_clusters) | | Describe your clusters from their given relevant TF-IDF terms using OpenAI |
| [explore\_database](/api-docs/prepare/enrich/explore_database) | | Explore database structure — generates a dataset of column metadata with statistics, relationships, and semantic descr… |
| [fetch\_apify\_dataset](/api-docs/prepare/enrich/fetch_apify_dataset) | | Fetch data from an existing Apify dataset |
| [fetch\_demographics\_es](/api-docs/prepare/enrich/fetch_demographics_es) | | Fetch Spanish demographic census data given a geographical location in each row |
| [fetch\_from\_sql](/api-docs/prepare/enrich/fetch_from_sql) | | Fetch data from a SQL database query and return it as a dataset |
| [fetch\_full\_contact\_domains](/api-docs/prepare/enrich/fetch_full_contact_domains) | | Enrich a dataset containing links (URLs) to companies' online presence using the FullContact service |
| [fetch\_full\_contact\_emails](/api-docs/prepare/enrich/fetch_full_contact_emails) | | Enrich a dataset containing email addresses with personal information using the *FullContact* service |
| [fetch\_google\_places](/api-docs/prepare/enrich/fetch_google_places) | | Fetch information about the most relevant places surrounding a location |
| [fetch\_google\_vision](/api-docs/prepare/enrich/fetch_google_vision) | | Analyze images given their URL using the Google Vision API |
| [fetch\_location](/api-docs/prepare/enrich/fetch_location) | | Extract formatted address, locality, area, state, country and geographical coordinates from one or more address columns |
| [fetch\_openreview](/api-docs/prepare/enrich/fetch_openreview) | | Fetch publications submitted to one or more conferences via [OpenReview](https://openreview.net/) |
| [fetch\_social\_shares](/api-docs/prepare/enrich/fetch_social_shares) | | Fetch the number of times a Url was shared on Facebook |
| [fetch\_twitter](/api-docs/prepare/enrich/fetch_twitter) | | Enriches a dataset containing tweets with information about their authors |
| [fetch\_twitter\_api\_io](/api-docs/prepare/enrich/fetch_twitter_api_io) | | Fetch tweets from Twitter/X using TwitterAPI.io advanced search |
| [fetch\_url\_content](/api-docs/prepare/enrich/fetch_url_content) | | Fetch the main text from a web URL, and return its title, author, content, excerpt and domain |
| [infer\_aspect\_polarity](/api-docs/prepare/enrich/infer_aspect_polarity) | | Extract aspect-based polarity from texts using OpenAI. Identifies entities mentioned in textsand separates them into p… |
| [infer\_aspect\_sentiment](/api-docs/prepare/enrich/infer_aspect_sentiment) | | Extract aspect-based sentiments from texts using OpenAI. Identifies entities mentioned in textswith clear positive or … |
| [infer\_gender](/api-docs/prepare/enrich/infer_gender) | | Try to infer a person's gender given a first name |
| [infer\_language](/api-docs/prepare/enrich/infer_language) | | Detect the language used for each text in the input column |
| [infer\_missing](/api-docs/prepare/enrich/infer_missing) | | Train and use a machine learning model to predict (impute) the missing values in a column |
| [infer\_missing\_with\_probs](/api-docs/prepare/enrich/infer_missing_with_probs) | | Train and use a machine learning model to predict (impute) the missing values in a column |
| [infer\_sentiment](/api-docs/prepare/enrich/infer_sentiment) | | Parse text and calculate the overall positive or negative sentiment polarity |
| [infer\_topics](/api-docs/prepare/enrich/infer_topics) | | Generate topics and subtopics for given texts using OpenAI. Infers a hierarchical topic structurefrom (a sample of) th… |
| [prompt\_ai](/api-docs/prepare/enrich/prompt_ai) | | Call OpenAI's models on each row of the dataset for a given prompt |
| [run\_apify\_actor](/api-docs/prepare/enrich/run_apify_actor) | | Run an Apify actor and fetch the resulting dataset |
| [zeroshot\_classify\_text](/api-docs/prepare/enrich/zeroshot_classify_text) | | Classify texts using custom labels/categories |
# infer_aspect_polarity
Source: https://docs.graphext.com/api-docs/prepare/enrich/infer_aspect_polarity
Extract aspect-based polarity from texts using OpenAI. Identifies entities mentioned in texts and separates them into positive and negative aspects based on expressed polarity. Returns two columns: one containing positively-mentioned entities and another containing negatively-mentioned entities. Optionally returns two additional columns with the reasons for the positive and negative classifications.
## Usage
The following examples show how the step can be used in a recipe.
Extract positive and negative aspects from a text column
```stan theme={null}
infer_aspect_polarity(ds.texts, {
"integration": "open-ai-1",
"model": "openai/gpt-4.1",
}) -> (ds.positive_aspects, ds.negative_aspects)
```
Extract aspects with their reasons
```stan theme={null}
infer_aspect_polarity(ds.texts, {
"integration": "open-ai-1",
"model": "openai/gpt-4.1",
}) -> (ds.positive_aspects, ds.negative_aspects, ds.positive_reasons, ds.negative_reasons)
```
Extract aspects with reasons and categories
```stan theme={null}
infer_aspect_polarity(ds.texts, {
"integration": "open-ai-1",
"model": "openai/gpt-4.1",
"aspect_categories": ["food", "service", "pricing"],
}) -> (ds.positive_aspects, ds.negative_aspects, ds.positive_reasons, ds.negative_reasons, ds.categories)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
infer_aspect_polarity(texts: category|text, {
"param": value,
...
}) -> (*aspects: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Column containing the texts to extract aspect polarity from.
Output columns for aspect polarity results. If two column names are provided, returns
positive and negative aspects. If four column names are provided, additionally returns
the reasons for the positive and negative classifications. If five column names are
provided and aspect\_categories is set, additionally returns the aggregated categories.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Associated integration.
AI Model.
AI model used for aspect-based polarity extraction. Each text is processed individually
to identify entities and classify them as positive or negative.
Values must be one of the following:
`openai/gpt-4.1` `openai/gpt-4.1-mini` `openai/gpt-4.1-nano` `openai/gpt-5` `openai/gpt-5-mini` `openai/gpt-5-nano` `openai/gpt-5.1` `openai/gpt-5.2`
Additional Instructions.
Additional instructions to guide the polarity extraction process. Use this to provide
domain-specific guidance or to focus on particular types of entities.
Aspect Categories.
Optional list of aspect categories to focus on during extraction. When provided, the model
will classify each extracted entity into one of these categories, and an additional
categories output column will be available. For example:
\["food", "service", "pricing", "cleanliness", "wait times"].
Each item in array.
Deduplicate Results.
Whether to deduplicate extracted entities and/or reasons. Can be false (no deduplication),
true (deduplicate reasons only), or a list specifying which fields to deduplicate
(e.g., \["entities", "reasons"]).
Each item in array.
Values must be one of the following:
* `entities`
* `reasons`
Deduplication Model.
AI model used for deduplication when deduplicate is enabled.
Values must be one of the following:
`openai/gpt-4.1` `openai/gpt-4.1-mini` `openai/gpt-4.1-nano` `openai/gpt-5` `openai/gpt-5-mini` `openai/gpt-5-nano` `openai/gpt-5.1` `openai/gpt-5.2`
Deduplication Batch Size.
Maximum number of entities per deduplication LLM call. Increase for larger datasets,
decrease if hitting context limits.
Values must be in the following range:
```javascript theme={null}
100 ≤ dedupe_batch_size ≤ 10000
```
Deduplication Instructions.
Additional instructions to guide the deduplication clustering process. Use this to
specify domain-specific clustering rules, e.g., "Keep ride-related complaints separate
from food-related complaints" or "Treat pricing complaints for different items as distinct".
Consolidate Clusters.
Whether to run an extra merge pass to consolidate similar clusters. This helps reduce
near-duplicate clusters even within a single batch.
API Parameters.
Additional parameters passed to the responses API.
# infer_gender
Source: https://docs.graphext.com/api-docs/prepare/enrich/infer_gender
Try to infer a person's gender given a first name.
Uses a machine learning model trained on a large database of names and the frequencies of associated genders.
## Usage
The following examples show how the step can be used in a recipe.
To use the default labels "male" and "female" in the resulting output simply use
```stan theme={null}
infer_gender(ds.first_name) -> (ds.gender)
```
To use labels "M" and "F" instead
```stan theme={null}
infer_gender(ds.first_name), "labels": {"male": "M", "female": "F"}) -> (ds.gender)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
infer_gender(first_name: category, {
"param": value,
...
}) -> (gender: sex)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Column containing first names.
Predicted gender for each name.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Labels for the male and female categories.
An object mapping the "male" and "female" categories to custom labels.
Label for the "male" category.
Label for the "female" category.
# infer_language
Source: https://docs.graphext.com/api-docs/prepare/enrich/infer_language
Detect the language used for each text in the input column.
Each language will be represented by its [ISO 639-1 language code](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes),
such as "en", "es", "it" for English, Spanish and Italian respectively.
## Usage
The following example shows how the step can be used in a recipe.
In most cases no special configuration should be necessary, so simply
```stan theme={null}
infer_language(ds.text) -> (ds.language)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
infer_language(text: text, {
"param": value,
...
}) -> (lang: category)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A text column to detect languages for.
A column identifying the language of each text using its two-letter
[ISO 639-1 language code](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Which model to use to detect languages.
Select from one of four model types (corresponding to specific Python libraries):
* `"lingua"`: [https://github.com/pemistahl/lingua-py](https://github.com/pemistahl/lingua-py)
* `"fasttext"`: [https://fasttext.cc/docs/en/language-identification.html](https://fasttext.cc/docs/en/language-identification.html)
* `"langdetect"`: [https://github.com/Mimino666/langdetect](https://github.com/Mimino666/langdetect)
* `"langid"`: [https://github.com/saffsd/langid.py](https://github.com/saffsd/langid.py).
Values must be one of the following:
* `lingua`
* `fasttext`
* `langdetect`
* `langid`
Whether to lowercase texts before detection.
Some models may be more sensitive than others if texts are in capital letters only, for example.
Minimum probability to assign a language for a particular text.
If the model used to infer the language is less sure about a language than this, the corresponding
text will be assigned the "undefined" language ("und"). Note that a reasonable value might depend
on the specific model used. Different models may produce different distributions of detection confidence.
Values must be in the following range:
```javascript theme={null}
0 ≤ min_probability ≤ 1
```
Restrict which languages can be be inferred.
Can be used to limit language detection to a smaller set if necessary. By default (when
not specifying this parameter, or when setting it to `true` or `null`), we restrict this
to the languages which we have spaCy models for, because this is the most common use of
language detection in Graphext (applying the correct spaCy language model to extract keywords
e.g.).
If set to `false`, will allow detection of all languages supported by the selected model.
If set to a list of ISO 639-1 codes, only these languages are detected (if supported by
the model).
Each item in array.
# infer_missing
Source: https://docs.graphext.com/api-docs/prepare/enrich/infer_missing
Train and use a machine learning model to predict (impute) the missing values in a column.
Non-missing values in the target column will be used to train a prediction model (a [Catboost](https://catboost.ai/) regressor or classifier), which then predicts (imputes) the missing values. Only simple numerical or categorical input data can be imputed.
## Usage
The following example shows how the step can be used in a recipe.
To automatically select the a model (classifier vs regressor) based on the kind of target
variable (numeric or categorical), simply use:
```stan theme={null}
infer_missing(ds, {"target": "incomplete_col"}) -> (ds.complete_col)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
infer_missing(ds: dataset, {
"param": value,
...
}) -> (predicted: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A dataset containing the column to be imputed as well as any other column to use as predictors in the model.
A column containing the predicted values for all rows.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Name of the column to impute.
The step will predict the missing values for this column, using rows in the dataset where the values are not missing to train the prediction model.
Predict non-missing values.
When set to true, all values are predicted. Set this param to false to maintain original values when they are not missing.
Confidence threshold.
Every prediction with probability strictly below this threshold will be set to NaN (missing).
Values must be in the following range:
```javascript theme={null}
0 ≤ threshold < 1
```
CatBoost configuration parameters.
You can check the official documentation for more details about Catboost's parameters [here](https://catboost.ai/en/docs/references/training-parameters/).
The maximum depth of the tree.
Values must be in the following range:
```javascript theme={null}
2 ≤ depth ≤ 16
```
Number of iterations.
The maximum number of trees that can be built when solving machine learning problems. When using other parameters that limit the number of iterations, the final number of trees may be less than the number specified in this parameter.
Values must be in the following range:
```javascript theme={null}
1 ≤ iterations < inf
```
Maximum cardinality of variables to be one-hot encoded.
Use one-hot encoding for all categorical features with a number of different values less than or equal to this value. Other variables will be target-encoded. Note that one-hot encoding is faster than the alternatives, so decreasing this value makes it more likely slower methods will be used. See [CatBoost details](https://catboost.ai/docs/concepts/algorithm-main-stages_cat-to-numberic.html) for further information.
Values must be in the following range:
```javascript theme={null}
2 ≤ one_hot_max_size < inf
```
The maximum number of features that can be combined when transforming categorical variables.
Each resulting combination consists of one or more categorical features and can optionally contain binary features in the following form: “numeric feature > value”.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_ctr_complexity ≤ 4
```
Coefficient at the L2 regularization term of the cost function.
Values must be in the following range:
```javascript theme={null}
0.0 < l2_leaf_reg < inf
```
The number of splits for numerical features.
Values must be in the following range:
```javascript theme={null}
1 ≤ border_count ≤ 65535
```
The amount of randomness to use for scoring splits.
Use this parameter to avoid overfitting the model. The value multiplies the variance of a random variable (with
zero mean) that is added to the score used to select splits when a tree is grown.
Values must be in the following range:
```javascript theme={null}
0 < random_strength < inf
```
The method for processing missing values in the input dataset.
Possible values:
* “Forbidden”:
Missing values are not supported, their presence is interpreted as an error.
* “Min”:
Missing values are processed as the minimum value (less than all other values) for the feature.
It is guaranteed that a split that separates missing values from all other values is considered
when selecting trees.
* “Max”:
Missing values are processed as the maximum value (greater than all other values) for the feature.
It is guaranteed that a split that separates missing values from all other values is considered when
selecting trees.
Using the Min or Max value of this parameter guarantees that a split between missing values and other
values is considered when selecting a new split in the tree.
Values must be one of the following:
* `Forbidden`
* `Min`
* `Max`
Boosting type.
Boosting scheme. Possible values are
* Ordered: Usually provides better quality on small datasets, but it may be slower than the Plain scheme.
* Plain: The classic gradient boosting scheme.
Values must be one of the following:
* `Ordered`
* `Plain`
Random subspace method.
The percentage of features to use at each split selection, when features are selected over again at random. The value `null` is equivalent to 1.0 (all features). You can set this to values \< 1.0 when the dataset has many features (e.g. > 20) to speed up training.
Values must be in the following range:
```javascript theme={null}
0 < rsm ≤ 1.0
```
The random seed used for training.
Whether and how to limit memory usage.
Select the maximum Ram used using strings like "2GB" or "100mb" (non case\_sensitive).
Toggle encoding of feature columns.
When enabled, Graphext will auto-convert any column types to the numeric type before
fitting the model. How this conversion is done can be configured using the `feature_encoder`
option below.
If disabled, any model trained in this step will assume that input data
is already in an appropriate format (e.g. numerical and not containing any missing values).
Configures encoding of feature columns.
By default (`null`), Graphext chooses automatically how to convert any column types the model
may not understand natively to a numeric type.
A configuration object can be passed instead to overwrite specific parameter values with respect
to their default values.
Numeric encoder.
Configures encoding of numeric features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Further parameters passed to the `scaler` function.
Details depend no the particular scaler used.
Boolean encoder.
Configures encoding of boolean features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Ordinal encoder.
Configures encoding of categorical features that have a natural order.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Category encoder.
May contain either a single configuration for all categorical variables, or two different configurations
for low- and high-cardinality variables. For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for categories with fewer than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Multilabel encoder.
Configures encoding of multivalued categorical features (variable length lists of categories,
or the semantic type `list[category]` for short). May contain either a single configuration for
all multilabel variables, or two different configurations for low- and high-cardinality variables.
For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for mulitabel columns with fewer than `cardinality_threshold` unique categories/labels.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Datetime encoder.
Configures encoding of datetime (timestamp) features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
A list of numerical components to extract.
Will create one numeric column for each component.
Each item in array.
Values must be one of the following:
`day` `dayofweek` `dayofyear` `hour` `minute` `month` `quarter` `season` `second` `week` `weekday` `weekofyear` `year`
A list of cyclical time features to extract.
"Cycles" are numerical transformations of features that should be represented on a circle. E.g. months,
ranging from 1 to 12, should be arranged such that 12 and 1 are next to each other, rather than on
opposite ends of a linear scale. We represent such cyclical time features on a circle by creating two
columns for each original feature: the sin and cos of the numerical feature after appropriate scaling.
Each item in array.
Values must be one of the following:
* `day`
* `dayofweek`
* `dayofyear`
* `hour`
* `month`
Whether to include the epoch as new feature (seconds since 01/01/1970).
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Embedding/vector encoder.
Configures encoding of multivalued numerical features (variable length lists of numbers, i.e. vectors, or the semantic type `list[number]` for short).
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Text encoder.
Configures encoding of text (natural language) features. Currently only allows
[Tf-Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) embeddings to represent texts. If you wish
to use other embeddings, e.g. semantic, Word2Vec etc., transform your text column first using
another step, and then use that result instead of the original texts.
Texts are *excluded* by default from the overall encoding of the dataset. See parameter
`include_text_features` below to active it.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Parameters to be passed to the text encoder (Tf-Idf parameters only for now).
See [scikit-learn's documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)
for detailed parameters and their explanation.
How many output features to generate.
The resulting Tf-Idf vectors will be reduced to these many dimensions (columns) using scikit-learn's
[truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
This performs a kind of [latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
By default we will reduce to 200 components.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_components ≤ 1024
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Whether to include or ignore text columns during the processing of input data.
Enabling this will convert texts to their TfIdf representation. Each text will be
converted to an N-dimensional vector in which each component measures the relative
"over-representation" of a specific word (or n-gram) relative to its overall
frequency in the whole dataset. This is disabled by default because it will
often be better to convert texts explicitly using a previous step, such as
`embed_text` or `embed_text_with_model`.
Configure model validation.
Allows evaluation of model performance via cross-validation with custom metrics. If not
specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [cross-validation]() to split the dataset. E.g. if `n_splits`
is 5, the dataset will be split into 5 equal-sized parts. For five iterations four parts will then
be used for training and the remaining part for testing. If `test_size` is a number between 0 and 1,
in contrast, validation is done using a [shuffle-split]() approach. Here, instead of splitting the data into
`n_splits` equal parts up front, in each iteration we randomize the data and sample a proportion equal
to `test_size` to use for evaluation and the remaining rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
null.
array.
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `explained_variance` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `neg_mean_squared_error` `neg_median_absolute_error` `neg_root_mean_squared_error` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `r2`
Configure hypertuning.
Configures the optimization of model hyper-parameters via cross-validated grid- or randomized search.
The parameter values to explore.
Allows tuning of any and all of the parameters that can be set also as constants in the
"params" attribute.
Keys in this object should be strings identifying parameter names, and values should be
*lists* of values to explore for that parameter. E.g. `"depth": [3, 5, 7]`.
List of depths values to explore.
The maximum depth of the tree.
Values must be in the following range:
```javascript theme={null}
2 ≤ Item ≤ 16
```
List of iterations values to explore.
Number of iterations.
The maximum number of trees that can be built when solving machine learning problems. When using other parameters that limit the number of iterations, the final number of trees may be less than the number specified in this parameter.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item < inf
```
List of values configuring max cardinality for one-hot encoding.
Maximum cardinality of variables to be one-hot encoded.
Use one-hot encoding for all categorical features with a number of different values less than or equal to this value. Other variables will be target-encoded. Note that one-hot encoding is faster than the alternatives, so decreasing this value makes it more likely slower methods will be used. See [CatBoost details](https://catboost.ai/docs/concepts/algorithm-main-stages_cat-to-numberic.html) for further information.
Values must be in the following range:
```javascript theme={null}
2 ≤ Item < inf
```
List of values configuring variable combination complexity.
The maximum number of features that can be combined when transforming categorical variables.
Each resulting combination consists of one or more categorical features and can optionally contain binary features in the following form: “numeric feature > value”.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item ≤ 4
```
List of leaf regularization strengths.
Coefficient at the L2 regularization term of the cost function.
Values must be in the following range:
```javascript theme={null}
0.0 < Item < inf
```
List of border counts.
The number of splits for numerical features.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item ≤ 65535
```
List of random strengths.
The amount of randomness to use for scoring splits.
Use this parameter to avoid overfitting the model. The value multiplies the variance of a random variable (with
zero mean) that is added to the score used to select splits when a tree is grown.
Values must be in the following range:
```javascript theme={null}
0 < Item < inf
```
Which search strategy to use for optimization.
Grid search explores all possible combinations of parameters specified in `params`.
Randomized search, on the other hand, randomly samples `iterations` parameter combinations
from the distributions specified in `params`.
Values must be one of the following:
* `grid`
* `random`
How many randomly sampled parameter combinations to test in randomized search.
Values must be in the following range:
```javascript theme={null}
1 < iterations < inf
```
Configure model validation.
Allows evaluation of model performance via cross-validation with custom metrics. If not
specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [cross-validation]() to split the dataset. E.g. if `n_splits`
is 5, the dataset will be split into 5 equal-sized parts. For five iterations four parts will then
be used for training and the remaining part for testing. If `test_size` is a number between 0 and 1,
in contrast, validation is done using a [shuffle-split]() approach. Here, instead of splitting the data into
`n_splits` equal parts up front, in each iteration we randomize the data and sample a proportion equal
to `test_size` to use for evaluation and the remaining rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
null.
array.
Each item in array.
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `explained_variance` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `neg_mean_squared_error` `neg_median_absolute_error` `neg_root_mean_squared_error` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `r2`
Seed for random number generator ensuring reproducibility.
Values must be in the following range:
```javascript theme={null}
0 ≤ seed < inf
```
# infer_missing_with_probs
Source: https://docs.graphext.com/api-docs/prepare/enrich/infer_missing_with_probs
Train and use a machine learning model to predict (impute) the missing values in a column.
Non-missing values in a categorical target column will be used to train a prediction model (a [Catboost](https://catboost.ai/) classifier), which then predicts (imputes) the missing values. The step produces two output columns: one containing predicted classes for all rows, and a second containing a probability for each predicted class.
## Usage
The following examples show how the step can be used in a recipe.
Predict missing categories and their probabilities
```stan theme={null}
infer_missing_with_probs(ds, {"target": "category_col"}) => (ds.predicted_class, ds.probability)
```
Predict all values with a confidence threshold of 0.5
```stan theme={null}
infer_missing_with_probs(ds, {"target": "category_col", "infer_all": true, "threshold": 0.5}) => (ds.predicted_class, ds.probability)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
infer_missing_with_probs(ds: dataset, {
"param": value,
...
}) -> (predicted: category, probs: number)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A dataset containing the column to be imputed as well as any other column to use as predictors in the model.
A column containing the predicted classes for all rows.
Probability estimate (of the predicted class only).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Name of the categorical column to impute.
The step will predict the missing (and non-missing) class labels for
this column, using rows in the dataset where the values are not missing to train the prediction model.
Predict non-missing values.
When set to true, all values are predicted. Set this param to false to maintain original values when they are not missing.
Confidence threshold.
Every prediction with probability strictly below this threshold will be set to NaN (missing).
Values must be in the following range:
```javascript theme={null}
0 ≤ threshold < 1
```
CatBoost configuration parameters.
You can check the official documentation for more details about Catboost's parameters [here](https://catboost.ai/en/docs/references/training-parameters/).
The maximum depth of the tree.
Values must be in the following range:
```javascript theme={null}
2 ≤ depth ≤ 16
```
Number of iterations.
The maximum number of trees that can be built when solving machine learning problems. When using other parameters that limit the number of iterations, the final number of trees may be less than the number specified in this parameter.
Values must be in the following range:
```javascript theme={null}
1 ≤ iterations < inf
```
Maximum cardinality of variables to be one-hot encoded.
Use one-hot encoding for all categorical features with a number of different values less than or equal to this value. Other variables will be target-encoded. Note that one-hot encoding is faster than the alternatives, so decreasing this value makes it more likely slower methods will be used. See [CatBoost details](https://catboost.ai/docs/concepts/algorithm-main-stages_cat-to-numberic.html) for further information.
Values must be in the following range:
```javascript theme={null}
2 ≤ one_hot_max_size < inf
```
The maximum number of features that can be combined when transforming categorical variables.
Each resulting combination consists of one or more categorical features and can optionally contain binary features in the following form: “numeric feature > value”.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_ctr_complexity ≤ 4
```
Coefficient at the L2 regularization term of the cost function.
Values must be in the following range:
```javascript theme={null}
0.0 < l2_leaf_reg < inf
```
The number of splits for numerical features.
Values must be in the following range:
```javascript theme={null}
1 ≤ border_count ≤ 65535
```
The amount of randomness to use for scoring splits.
Use this parameter to avoid overfitting the model. The value multiplies the variance of a random variable (with
zero mean) that is added to the score used to select splits when a tree is grown.
Values must be in the following range:
```javascript theme={null}
0 < random_strength < inf
```
The method for processing missing values in the input dataset.
Possible values:
* “Forbidden”:
Missing values are not supported, their presence is interpreted as an error.
* “Min”:
Missing values are processed as the minimum value (less than all other values) for the feature.
It is guaranteed that a split that separates missing values from all other values is considered
when selecting trees.
* “Max”:
Missing values are processed as the maximum value (greater than all other values) for the feature.
It is guaranteed that a split that separates missing values from all other values is considered when
selecting trees.
Using the Min or Max value of this parameter guarantees that a split between missing values and other
values is considered when selecting a new split in the tree.
Values must be one of the following:
* `Forbidden`
* `Min`
* `Max`
Boosting type.
Boosting scheme. Possible values are
* Ordered: Usually provides better quality on small datasets, but it may be slower than the Plain scheme.
* Plain: The classic gradient boosting scheme.
Values must be one of the following:
* `Ordered`
* `Plain`
Random subspace method.
The percentage of features to use at each split selection, when features are selected over again at random. The value `null` is equivalent to 1.0 (all features). You can set this to values \< 1.0 when the dataset has many features (e.g. > 20) to speed up training.
Values must be in the following range:
```javascript theme={null}
0 < rsm ≤ 1.0
```
The random seed used for training.
Whether and how to limit memory usage.
Select the maximum Ram used using strings like "2GB" or "100mb" (non case\_sensitive).
Toggle encoding of feature columns.
When enabled, Graphext will auto-convert any column types to the numeric type before
fitting the model. How this conversion is done can be configured using the `feature_encoder`
option below.
If disabled, any model trained in this step will assume that input data
is already in an appropriate format (e.g. numerical and not containing any missing values).
Configures encoding of feature columns.
By default (`null`), Graphext chooses automatically how to convert any column types the model
may not understand natively to a numeric type.
A configuration object can be passed instead to overwrite specific parameter values with respect
to their default values.
Numeric encoder.
Configures encoding of numeric features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Further parameters passed to the `scaler` function.
Details depend no the particular scaler used.
Boolean encoder.
Configures encoding of boolean features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Ordinal encoder.
Configures encoding of categorical features that have a natural order.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Category encoder.
May contain either a single configuration for all categorical variables, or two different configurations
for low- and high-cardinality variables. For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for categories with fewer than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `MostFrequent`
* `Const`
* `None`
Maximum number of unique categories to encode.
Only the N-1 most common categories will be encoded, and the rest will be grouped into a single
"Others" category.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_categories < inf
```
How to encode categories.
Values must be one of the following:
`OneHot` `Label` `Ordinal` `Binary` `Frequency` `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
Multilabel encoder.
Configures encoding of multivalued categorical features (variable length lists of categories,
or the semantic type `list[category]` for short). May contain either a single configuration for
all multilabel variables, or two different configurations for low- and high-cardinality variables.
For further details pick one of the two options below.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Condition for application of low- or high-cardinality configuration.
Number of unique categories below which the `low_cardinality` configuration is used,
and above which the `high_cardinality` configuration is used.
Values must be in the following range:
```javascript theme={null}
3 ≤ cardinality_treshold < inf
```
Low cardinality configuration.
Used for mulitabel columns with fewer than `cardinality_threshold` unique categories/labels.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
High cardinality configuration.
Used for categories with more than `cardinality_threshold` unique categories.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to encode categories/labels in multilabel (list\[category]) columns.
Values must be one of the following:
* `Binarizer`
* `TfIdf`
* `None`
Maximum number of categories/labels to encode.
If a number is provided, the result of the encoding will be reduced to these many dimensions (columns)
using scikit-learn's [truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
When applied together with (after a) Tf-Idf encoding, this performs a kind of
[latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
Values must be in the following range:
```javascript theme={null}
2 ≤ max_categories < inf
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Datetime encoder.
Configures encoding of datetime (timestamp) features.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
A list of numerical components to extract.
Will create one numeric column for each component.
Each item in array.
Values must be one of the following:
`day` `dayofweek` `dayofyear` `hour` `minute` `month` `quarter` `season` `second` `week` `weekday` `weekofyear` `year`
A list of cyclical time features to extract.
"Cycles" are numerical transformations of features that should be represented on a circle. E.g. months,
ranging from 1 to 12, should be arranged such that 12 and 1 are next to each other, rather than on
opposite ends of a linear scale. We represent such cyclical time features on a circle by creating two
columns for each original feature: the sin and cos of the numerical feature after appropriate scaling.
Each item in array.
Values must be one of the following:
* `day`
* `dayofweek`
* `dayofyear`
* `hour`
* `month`
Whether to include the epoch as new feature (seconds since 01/01/1970).
Whether and how to impute (replace/fill) missing values.
Values must be one of the following:
* `Mean`
* `Median`
* `MostFrequent`
* `Const`
* `None`
Whether and how to scale the final numerical values (across a single column).
Values must be one of the following:
* `Standard`
* `Robust`
* `KNN`
* `None`
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Embedding/vector encoder.
Configures encoding of multivalued numerical features (variable length lists of numbers, i.e. vectors, or the semantic type `list[number]` for short).
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Text encoder.
Configures encoding of text (natural language) features. Currently only allows
[Tf-Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) embeddings to represent texts. If you wish
to use other embeddings, e.g. semantic, Word2Vec etc., transform your text column first using
another step, and then use that result instead of the original texts.
Texts are *excluded* by default from the overall encoding of the dataset. See parameter
`include_text_features` below to active it.
Toggle the addition of a column using 0s and 1s to indicate where an input column contained missing values.
Parameters to be passed to the text encoder (Tf-Idf parameters only for now).
See [scikit-learn's documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)
for detailed parameters and their explanation.
How many output features to generate.
The resulting Tf-Idf vectors will be reduced to these many dimensions (columns) using scikit-learn's
[truncated SVD](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html).
This performs a kind of [latent semantic analysis](https://en.wikipedia.org/wiki/Latent_semantic_analysis).
By default we will reduce to 200 components.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_components ≤ 1024
```
How to scale the encoded (numerical columns).
Values must be one of the following:
* `Euclidean`
* `KNN`
* `Norm`
* `None`
Whether to include or ignore text columns during the processing of input data.
Enabling this will convert texts to their TfIdf representation. Each text will be
converted to an N-dimensional vector in which each component measures the relative
"over-representation" of a specific word (or n-gram) relative to its overall
frequency in the whole dataset. This is disabled by default because it will
often be better to convert texts explicitly using a previous step, such as
`embed_text` or `embed_text_with_model`.
Configure model validation.
Allows evaluation of model performance via cross-validation with custom metrics. If not
specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [cross-validation]() to split the dataset. E.g. if `n_splits`
is 5, the dataset will be split into 5 equal-sized parts. For five iterations four parts will then
be used for training and the remaining part for testing. If `test_size` is a number between 0 and 1,
in contrast, validation is done using a [shuffle-split]() approach. Here, instead of splitting the data into
`n_splits` equal parts up front, in each iteration we randomize the data and sample a proportion equal
to `test_size` to use for evaluation and the remaining rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
null.
array.
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `explained_variance` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `neg_mean_squared_error` `neg_median_absolute_error` `neg_root_mean_squared_error` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `r2`
Configure hypertuning.
Configures the optimization of model hyper-parameters via cross-validated grid- or randomized search.
The parameter values to explore.
Allows tuning of any and all of the parameters that can be set also as constants in the
"params" attribute.
Keys in this object should be strings identifying parameter names, and values should be
*lists* of values to explore for that parameter. E.g. `"depth": [3, 5, 7]`.
List of depths values to explore.
The maximum depth of the tree.
Values must be in the following range:
```javascript theme={null}
2 ≤ Item ≤ 16
```
List of iterations values to explore.
Number of iterations.
The maximum number of trees that can be built when solving machine learning problems. When using other parameters that limit the number of iterations, the final number of trees may be less than the number specified in this parameter.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item < inf
```
List of values configuring max cardinality for one-hot encoding.
Maximum cardinality of variables to be one-hot encoded.
Use one-hot encoding for all categorical features with a number of different values less than or equal to this value. Other variables will be target-encoded. Note that one-hot encoding is faster than the alternatives, so decreasing this value makes it more likely slower methods will be used. See [CatBoost details](https://catboost.ai/docs/concepts/algorithm-main-stages_cat-to-numberic.html) for further information.
Values must be in the following range:
```javascript theme={null}
2 ≤ Item < inf
```
List of values configuring variable combination complexity.
The maximum number of features that can be combined when transforming categorical variables.
Each resulting combination consists of one or more categorical features and can optionally contain binary features in the following form: “numeric feature > value”.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item ≤ 4
```
List of leaf regularization strengths.
Coefficient at the L2 regularization term of the cost function.
Values must be in the following range:
```javascript theme={null}
0.0 < Item < inf
```
List of border counts.
The number of splits for numerical features.
Values must be in the following range:
```javascript theme={null}
1 ≤ Item ≤ 65535
```
List of random strengths.
The amount of randomness to use for scoring splits.
Use this parameter to avoid overfitting the model. The value multiplies the variance of a random variable (with
zero mean) that is added to the score used to select splits when a tree is grown.
Values must be in the following range:
```javascript theme={null}
0 < Item < inf
```
Which search strategy to use for optimization.
Grid search explores all possible combinations of parameters specified in `params`.
Randomized search, on the other hand, randomly samples `iterations` parameter combinations
from the distributions specified in `params`.
Values must be one of the following:
* `grid`
* `random`
How many randomly sampled parameter combinations to test in randomized search.
Values must be in the following range:
```javascript theme={null}
1 < iterations < inf
```
Configure model validation.
Allows evaluation of model performance via cross-validation with custom metrics. If not
specified, will by default perform 5-fold cross-validation with automatically selected
metrics.
Number of train-test splits to evaluate the model on.
Will split the dataset into training and test set `n_splits` times, train on the former
and evaluate on the latter using specified or automatically selected `metrics`.
What proportion of the data to use for testing in each split.
If `null` or not provided, will use [cross-validation]() to split the dataset. E.g. if `n_splits`
is 5, the dataset will be split into 5 equal-sized parts. For five iterations four parts will then
be used for training and the remaining part for testing. If `test_size` is a number between 0 and 1,
in contrast, validation is done using a [shuffle-split]() approach. Here, instead of splitting the data into
`n_splits` equal parts up front, in each iteration we randomize the data and sample a proportion equal
to `test_size` to use for evaluation and the remaining rows for training.
Values must be in the following range:
```javascript theme={null}
0 < test_size < 1
```
One or more metrics/scoring functions to evaluate the model with.
When none is provided, will measure default metrics appropriate for the prediction task
(classification vs. regression determined from model or type of target column). See
[sklearn model evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
for further details.
null.
array.
Each item in array.
Each item in array.
Values must be one of the following:
`accuracy` `balanced_accuracy` `explained_variance` `f1_micro` `f1_macro` `f1_samples` `f1_weighted` `neg_mean_squared_error` `neg_median_absolute_error` `neg_root_mean_squared_error` `precision_micro` `precision_macro` `precision_samples` `precision_weighted` `recall_micro` `recall_macro` `recall_samples` `recall_weighted` `r2`
Seed for random number generator ensuring reproducibility.
Values must be in the following range:
```javascript theme={null}
0 ≤ seed < inf
```
# infer_sentiment
Source: https://docs.graphext.com/api-docs/prepare/enrich/infer_sentiment
Parse text and calculate the overall positive or negative sentiment polarity.
Polarity is measured on the normalized scale \[-1, 1]. The method used here is rather naïve. It simply looks
up each word in the text in a "polarity lexicon", which assigns each emotionally charged word a numeric
score. The individual scores are then simply averaged across the whole text. This will hence not account
for contexts involving irony, sarcasm, or even simple negations.
## Usage
The following examples show how the step can be used in a recipe.
To detect the sentiment for languages supported by default, use:
```stan theme={null}
infer_sentiment(ds.text, ds.lang) -> (ds.sentiment)
```
To only process those languages used in at least 1% of the input texts:
```stan theme={null}
infer_sentiment(ds.text, ds.lang, {"min_lang_docs": 0.01}) -> (ds.sentiment)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
infer_sentiment(text: text, *lang: category, {
"param": value,
...
}) -> (sentiment: number)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A text column to infer sentiment polarities for.
An (optional) column identifying the languages of the corresponding texts. It is used to identify the correct model (spaCy)
to use for each text. If the dataset doesn't contain such a column yet, it can be created using the `infer_language` step.
Ideally, languages should be expressed as two-letter
[ISO 639-1 language codes](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes), such as "en", "es" or "de" for
English, Spanish or German respectively. We also detect fully spelled out names such as "english", "German", "allemande"
etc., but it is not guaranteed that we will recognize all possible spellings correctly always, so ISO codes should be
preferred.
Alternatively, if all texts are in the same language, it can be identified with the `language` *parameter* instead.
A column containing the overall sentiment polarity for each input text.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Whether to enable support for additional languages.
By default, Arabic ("ar"), Catalan ("ca"), Basque ("eu"), and Turkish ("tu") are not enabled,
since they're supported only by a different class of language models (stanfordNLP's Stanza)
that is much slower than the rest. This parameter can be used to enable them.
Minimum number (or proportion) of texts to include a language in processing.
Any texts in a language with fewer documents than these will be ignored. Can be useful to speed up
processing when there is noise in the input languages, and when ignoring languages with a small number of
documents only is acceptable. Values smaller than 1 will be interpreted as a *proportion* of all texts, and
values greater than or equal to 1 as an *absolute number* of documents.
number.
Values must be in the following range:
```javascript theme={null}
0 < {_} < 1
```
integer.
Values must be in the following range:
```javascript theme={null}
1 ≤ {_} < inf
```
The language of inputs texts.
If all texts are in the same language, it can be specified here instead of passing it as an input column. The language will be used to identify the correct spaCy model to parse and analyze the texts. For allowed values, see the comment regarding the `lang` column above.
# infer_topics
Source: https://docs.graphext.com/api-docs/prepare/enrich/infer_topics
Generate topics and subtopics for given texts using OpenAI.
## Usage
The following example shows how the step can be used in a recipe.
Generate topics for a given text column
```stan theme={null}
infer_topics(ds.texts, {
"integration": "open-ai-1",
"n_topics": 10,
"n_subtopics": 5,
"inference_model": "openai/gpt-4.1",
"assignment_model": "openai/gpt-4.1-mini",
}) ->(ds.topic, ds.subtopic)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
infer_topics(texts: category|text, {
"param": value,
...
}) -> (topic: category, subtopic: category)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Column containing the texts to infer topics from.
Inferred topic for each text.
Inferred subtopic for each text.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Associated integration.
Number of Topics.
Maximum approximate number of topics to infer.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_topics ≤ 20
```
Number of Subtopics.
Maximum approximate number of subtopics to infer per topic.
Values must be in the following range:
```javascript theme={null}
2 ≤ n_subtopics ≤ 10
```
Number of Samples.
Maximum number of text samples to use for topic extraction. More texts consume more tokens and increase cost.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_samples ≤ 10000
```
Multi-topic Assignment.
Whether to allow assigning multiple topics to each text. If enabled, the output columns will contain lists of
topics/subtopics instead of single values.
Inference AI Model.
AI model used to infer topic hierarchy. This model will receive all sample texts, so potentially
a large context, and needs to be reasonably capable to generate a well-structured topic hierarchy
(no repeated or similar topics etc.).
Values must be one of the following:
`openai/gpt-4.1` `openai/gpt-4.1-mini` `openai/gpt-4.1-nano` `openai/gpt-5` `openai/gpt-5-mini` `openai/gpt-5-nano` `openai/gpt-5.1` `openai/gpt-5.2`
Assignment AI Model.
AI model used to assign topic and subtopics to each text (row). This model will receive individual texts
along with the inferred topic hierarchy, so it can be a smaller model focused on classification.
Values must be one of the following:
`openai/gpt-4.1` `openai/gpt-4.1-mini` `openai/gpt-4.1-nano` `openai/gpt-5` `openai/gpt-5-mini` `openai/gpt-5-nano` `openai/gpt-5.1` `openai/gpt-5.2`
Inference Parameters.
Additional parameters passed to the responses API for the inference call.
Assignment Parameters.
Additional parameters passed to the responses API for the assignment call.
# prompt_ai
Source: https://docs.graphext.com/api-docs/prepare/enrich/prompt_ai
Call OpenAI's models on each row of the dataset for a given prompt.
Use any of OpenAI's models on a row-by-row basis. This step doesn't feed the whole dataset into the model, so you won't be able to
perform operations that require more than one row at a time.
It can be used to perform a variety of tasks. Keep in mind that OpenAI's models are generative AI technologies, and thus can give incorrect responses.
It comes with a predefined budget of 5 \$USD, which will prevent the step from executing if it will cost over that budget.
It is advised that you use a filter step first to test the prompt out on a few rows, then launch it on the whole dataset.
Keep in mind our budget is a rough estimate, if you're concerned about cost you should set limits on OpenAI's side.
Your prompt will be configured by using two parameters: 'prompt' and 'response\_format'.
prompt is a text field while response\_format allows you to specify a JSON format for the model's
response, in the format of `{[expected_column]: "description"}`.
Both in the prompt and the response format descriptions you may refer to the row's attributes by using
`${attribute_name}`. Check the examples and parameter documentation below for more information.
???+ info "API integration"
To use this step your team needs to have the *OpenAI* integration configured in Graphext. The corresponding credentials
are required to connect to a third-party API. You can configure API integrations following the `INTEGRATIONS` or `ADD INTEGRATION`
link in the top-left corner of your Team's page, selecting `API keys`, and then the name of the desired third-party service.
First, create an OpenAI account or sign in.
Next, navigate to the API key page and "Create new secret key", optionally naming the key.
Make sure to save this somewhere safe and do not share it with anyone.
Optionally, you can specify the organization the key belongs to.
On [OpenAI](https://platform.openai.com/)'s' page, you can set general budgets for your api key and other settings that may interest you.
## Usage
The following examples show how the step can be used in a recipe.
Specify model
```stan theme={null}
prompt_ai(ds[["Local Address"]], { # contains column 'Local Address'
"integration": "MY_INTEGRATION_ID",
"model": {
"id": "gpt-4.1-mini",
"temperature": 0.2
},
"prompt": "What is the country for ${Local Address}"
}) -> (ds.country)
```
Get attributes from disneyland reviews
```stan theme={null}
prompt_ai(ds[["Review_Text"]],
{
"integration": "open-ai-1-70",
"prompt": "The following is a review from Disneyland. I want you to extract the topics mentioned, the Names of the Disney Characters mentioned and the Names of rides mentioned in this paragraph: '${Review_Text}'. If you do not find or recognize any name of people, company, or rides, simply do not answer anything. NEVER ANSWER WITH 'NULL' VALUE. IMPORTANT: DO NOT ANSWER ANYTHING ELSE IN ANY OTHER CIRCUMSTANCE. DO NOT ANSWER ANYTHING ELSE APART FROM THE JSON",
"model": {
"id": "gpt-4.1-nano"
},
"response_format": {
"topics": "topics mentioned",
"names_of_characters": "names of Disney Characters",
"names_of_rides": "names of rides"
},
"force_format": {
"topics": ["fun", "children", "ride", "food"]
},
"out_types": {
"topics": "list[category]"
}
}) -> (ds.topics,
ds.names_of_characters,
ds.names_of_rides)
```
Classify Tweets
```stan theme={null}
prompt_ai(ds[["authorName", "text"]],
{
"integration": "victoriano-apikey",
"budget": 15,
"model": {
"id": "gpt-4.1-mini",
"temperature": 0.2
},
"prompt": "Classify the following tweet text if it implicitly: criticizes, benefits, is neutral, or is unrelated to each of the main political parties in Spain or any of their members and leaders: ${text} considering the bias of the medium that wrote it with the medium's name: ${authorName}",
"response_format": {
"Clasificacion_PP": "classify the tweet text into only one of these 4 categories related to the Partido Popular (PP), its leader (Álberto Nuñez Feijoo), or any of its members: criticizes PP, benefits PP, neutral for PP, does not mention PP",
"Clasificacion_PSOE": "classify the tweet text into only one of these 4 categories related to the Spanish Socialist Workers' Party (PSOE), its leader (Pedro Sánchez), or any of its members: criticizes PSOE, benefits PSOE, neutral for PSOE, does not mention PSOE",
"Clasificacion_VOX": "classify the tweet text into only one of these 4 categories related to the VOX party, its leader (Santiago Abascal), or any of its members: criticizes VOX, benefits VOX, neutral for VOX, does not mention VOX",
"Clasificacion_SUMAR": "classify the tweet text into only one of these 4 categories related to the SUMAR party, its leader (Yolanda Díez), or any of its members: criticizes SUMAR, benefits SUMAR, neutral for SUMAR, does not mention SUMAR",
"media_bias": "classify the bias of the medium as: right, center, left"
},
"out_types": {
"Clasificacion_PP": "category",
"Clasificacion_PSOE": "category",
"Clasificacion_VOX": "category",
"Clasificacion_SUMAR": "category",
"media_bias": "category"
}
}) -> (ds.Clasificacion_PP,
ds.Clasificacion_PSOE,
ds.Clasificacion_VOX,
ds.Clasificacion_SUMAR,
ds.media_bias)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
prompt_ai(ds: dataset, {
"param": value,
...
}) -> (*outputs: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Dataset to enrich. Make sure it contains the necessary columns.
Number of columns to specify. By default it's set as only one column, of type category.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Main prompt for the API call.
The main body of instructions you wish to perform.
Associated integration.
Prompt instructions for each output column.
Further prompt instructions for each output column.
One or more additional parameters.
Values allowed in each output column.
If provided, values in each column will be restricted.
One or more additional parameters.
Each item in array.
Types for the output column(s).
Desired types for each output column. By default, they will all be categories.
One or more additional parameters.
Values must be one of the following:
`category` `date` `number` `boolean` `url` `sex` `text` `list[number]` `list[category]` `list[url]` `list[boolean]` `list[date]`
Model Configuration.
Configuration for OpenAI's model.
OpenAI model to choose.
Values must be one of the following:
`gpt-4.1` `gpt-4.1-mini` `gpt-4.1-nano` `gpt-5-mini` `gpt-5-nano` `o4-mini`
Temperature. Higher means more creativity, but also makes the model more likely to hallucinate. Lower temperature yields more deterministic results. Ignored for reasoning models (gpt-5-mini, gpt-5-nano, o4-mini).
Values must be in the following range:
```javascript theme={null}
0 ≤ temperature ≤ 1
```
Budget.
If present, the step will not execute if estimated input token cost exceeds this amount in USD.
If max\_out\_tokens is not set, we will minimum of the cost. If it is set, we will give a ceiling.
Actual cost may vary depending on a number of factors like your OpenAI plan. Check your plan before executing.
Maximum output tokens.
If set, each individual response will add to at most this amount. Allows for a budget theorical ceiling to be calculated before executing.
Size of concurrent request at a time.
Lowering this if you have very low rate limits in your plan might prevent empty responses.
Values must be in the following range:
```javascript theme={null}
1 ≤ batch_size ≤ 1000
```
Timeout for requests to OpenAI.
Values must be in the following range:
```javascript theme={null}
1 ≤ timeout < inf
```
# zeroshot_classify_text
Source: https://docs.graphext.com/api-docs/prepare/enrich/zeroshot_classify_text
Classify texts using custom labels/categories.
In contrast with [`classify_text`](https://docs.graphext.com/api-docs/prepare/enrich/text/classify_text/),
this step doesn't require a model specifically trained with the given labels. Any model from the
[Hugging Face hub](https://huggingface.co/models) that is compatible with their
[zeroshot classification pipeline](https://huggingface.co/transformers/master/main_classes/pipelines.html#zeroshotclassificationpipeline)
can be used here. By default this is the (English) [`valhalla/distilbart-mnli-12-3`](https://huggingface.co/valhalla/distilbart-mnli-12-3),
for a good trade-off between model size and accuracy. If a multilingual model is needed
you could try e.g. [`joeddav/xlm-roberta-large-xnli`](https://huggingface.co/joeddav/xlm-roberta-large-xnli/).
Note that we do not validate the model name before executing it, so make sure it
corresponds to an existing model in the hub, otherwise the step will fail.
## Usage
The following examples show how the step can be used in a recipe.
E.g., to classify English texts into the three topics `sport`, `politics` and `business`:
```stan theme={null}
zeroshot_classify_text(ds.text, {"labels": ["sport", "politics", "business"]}) -> (ds.topic)
```
Or to try and infer the sentiment of texts in multiple languages:
```stan theme={null}
zeroshot_classify_text(ds.review, {
"labels": ["positive", "negative"],
"template": "The sentiment of this review is {}.",
"model": "joeddav/xlm-roberta-large-xnli"
}) -> (ds.review_sentiment)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
zeroshot_classify_text(text: text, {
"param": value,
...
}) -> (class: category|list[category])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column of texts to classify.
The inferred class of each text. The labels of individual categories are those passed in using the `labels`
parameter below. Depending on whether multilabel classification is activated or not, the output will be
either a simple categorical, or a multilabel categorical column (containing list of categories).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
A list of labels/categories to automatically assign to each text.
This can be somewhat of a black art. As a simple, if perhaps obvious heuristic,
the fewer and less ambiguous the selected categories the faster and most
probably accurate the resulting classification. As the number and ambiguity of
categories increases one can expect less precise results.
Each item in array.
The name of a model.
This should be the full name (including the organization if applicable) of a model in the
[Hugging Face model hub](https://huggingface.co/models). You can copy it by clicking on the
icon next to the model's name on its dedicated web page.
Note that for now Hugging Face only supports models trained on NLI (natural language inference)
tasks in their zeroshot pipeline. These can be recognized usually by mentioning `nli`, `mnli`,
or `xnli` in their name. For further details on zeroshot learning using NLI models see
e.g. [here](https://joeddav.github.io/blog/2020/05/29/ZSL.html#Classification-as-Natural-Language-Inference).
Also, note that if the name doesn't correspond to a model existing in the hub the step will fail.
* joeddav/xlm-roberta-large-xnli
* facebook/bart-large-mnli
A custom hypothesis template.
Hugging Face's NLI-based zeroshot pipeline essentially converts each label into a whole phrase,
and then compares texts againt these phrases to see whether the phrase "agrees" with or "contradicts"
each text. The template parameter can be used to determine *how* a label is converted into a
phrase. The default phrase is `"This text is {}."`, where the curly braces are then replaced
with each label.
If you have texts in a specific language (and if you're using a model appropriate for that single language),
you should probably provide a corresponding template in that language. If you have texts in
mixed languages (and specify a multilingual model), the default template should be fine.
You may also consider using alternative templates specific for your task. E.g. if you're trying to
classify the overall sentiment of product reviews, you may try a template like
`"The sentiment of this review is {}."` (e.g. combined with `"labels": ["positive", "negative"]`).
Whether to allow multiple labels/classes per text.
If this parameter is `false` (default), only the label for the class with the highest probability
will be returned.
If it is `true`, each class will be assigned a probability between 0 and 1. The result will
then contain a list of labels corresponding to all classes with probabilities greater than the
threshold `min_prob` (see below). The classes will be returned in the form of ordered lists,
with the first element being the label of the class with the highest probability.
Only return labels for classes with probability greater than this value.
In single label classification, if even the most probable class falls below this threshold, a missing value
will be returned instead of a label.
When performing multilabel classification, any classes with probabilities below this threshold will simply
be removed from the list of labels in each row. A value of `null` (default), `0.0`, or simply not specifying
this parameter will disable filtering of categories. In this case, the result will contain all classes/labels
for each row, ordered by probability in descending order.
How many texts to process simultaneously.
May get ignored when running on CPU.
Values must be in the following range:
```javascript theme={null}
1 ≤ batch_size ≤ 64
```
# filter_containing
Source: https://docs.graphext.com/api-docs/prepare/filter/filter_containing
Filter rows containing any or all of a number of specified values.
Includes or excludes rows of the input datset based on the values of a selected text or list column. Depending on the
configuration, if the column contains any or all of the specified values, the corresponding rows will be kept or dropped
in the output dataset.
"Containment" here means texts in a text column containing one or more specified substrings (words), or lists in a
list column containing one or more elements matching the specified values. See below for illustrative examples.
## Usage
The following examples show how the step can be used in a recipe.
E.g., to keep only those rows whose values in the "address" column contain the text string "Madrid":
```stan theme={null}
filter_containing(ds, {"column": "address", "values": ["Madrid"]}) -> (ds_filtered)
```
Or, given a dataset with the column "jobs", containing lists of one or more job categories in each row, to keep only those rows where the list includes the word "journalist" (ignoring the letter case, i.e. upper or lower case):
```stan theme={null}
filter_containing(ds, {
"column": "jobs",
"values": ["journalist"],
"case_sensitive": false
}) -> (ds_filtered)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
filter_containing(ds_in: dataset, {
"param": value,
...
}) -> (ds_out: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An input dataset to filter.
A dataset containing the same columns as the input dataset, but including or excluding the matched rows.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Name of column to be matched against the specified `values`.
Values to be matched in each row to decide its inclusion or exclusion.
May be a single value or a list of values to be matched.
Each item in array.
* the
* \['the', 'cat']
* 2
* \[2, 3]
If `true`, matching rows will be excluded from the output dataset.
I.e., only rows *not* containing the specified values will be returned.
Rows must contain *all* specified value to pass filter, rather than *any*.
Text values must match case to pass filter.
# filter_duplicate_nodes
Source: https://docs.graphext.com/api-docs/prepare/filter/filter_duplicate_nodes
Remove duplicate nodes in a network.
For each pair of nodes connected by a link that indicates a similarity greater than a specified threshold, keeps only
one of the two nodes and rewires the deleted node's incoming and outgoing links to point to the "surviving" node.
## Usage
The following example shows how the step can be used in a recipe.
To de-duplicate pairs of nodes with a link weight (similarity) greater than 0.9
```stan theme={null}
filter_duplicate_nodes(network, {
"duplicate_threshold": 0.9
}) -> (network_filtered)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
filter_duplicate_nodes(network: dataset, {
"param": value,
...
}) -> (network_flt: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
dataset containing the *nodes* (rows) to de-duplicate and the *links* between nodes of the input dataset.
A new dataset containing the same columns as the input `data`, but without duplicate rows and having connections rewired such that none
points to a deleted node.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Similarity threshold for candidate nodes to be eliminated.
Any node linked to another node with a weight (usually similarity) greater than this value
will be eliminated. Default (`null`) corresponds to positive infinity (no de-duplication).
# filter_duplicates
Source: https://docs.graphext.com/api-docs/prepare/filter/filter_duplicates
Filter duplicate rows, keeping the first or last of each set of duplicates found only.
## Usage
The following example shows how the step can be used in a recipe.
To keep only the first row amongst a set of duplicates, identifying duplicates by inspecting
values in columns "address" and "name"
```stan theme={null}
filter_duplicates(ds, {"columns": ["address", "name"], "keep": "first"}) -> (ds_filtered)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
filter_duplicates(ds_in: dataset, {
"param": value,
...
}) -> (ds_out: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An input dataset to filter.
A dataset containing the same columns as the input dataset but including or excluding the matched rows.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Names of columns used to detect and filter rows containing duplicate values.
If not provided, will inspect all columns. Note that multivalued columns, i.e. those containing lists
of values will always be ignored when searching for duplicates (but will be included in the result).
Each item in array.
Which of a duplicate set of rows to keep in the result.
Specifically, whether to keep the first or last row amongst the duplicates.
Values must be one of the following:
* `first`
* `last`
if `true`, inverts the row selection.
I.e., only rows being duplicates (in the selected columns) will be included in the resulting dataset.
Row sorting before de-duplication.
E.g. when the order of first or last duplicate to retain depends on other variables. If not configured,
no sorting will be performed.
Sort column name(s).
These column(s) will be used to sort the dataset before de-duplication (if multiple, in specified order).
null.
string.
array.
Each item in array.
Whether to sort in ascending order (or in descending order if false).
If an array, must have the same length as `by` and specify the sort order for each column.
Each item in array.
# filter_missing
Source: https://docs.graphext.com/api-docs/prepare/filter/filter_missing
Filter rows based on missing values in one or more columns.
By default keeps only those rows where values in selected columns are not missing (non-NaNs). Using the `exclude`
parameter, the row selection can be inverted, such that only rows with missing values in selected rows
will be returned.
## Usage
The following example shows how the step can be used in a recipe.
To keep only those rows where neither "address" nor "name" is missing
```stan theme={null}
filter_missing(ds, {"columns": ["address", "name"]}) -> (ds_filtered)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
filter_missing(ds_in: dataset, {
"param": value,
...
}) -> (ds_out: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An input dataset to filter.
A dataset containing the same columns as the input dataset but including or excluding the matched rows.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Names of columns used to detect and filter rows containing missing values.
Each item in array.
if `true`, rows with *non*-missing values will be excluded.
I.e., only rows containing missing values in the selected columns will be included in the resulting dataset.
# filter_range
Source: https://docs.graphext.com/api-docs/prepare/filter/filter_range
Filter rows based on the numeric values in a given column.
Keeps or drops rows where numeric values fall within a desired range, i.e. are greater than a certain minimum,
and/or smaller than a maximum value.
## Usage
The following examples show how the step can be used in a recipe.
The following example creates a new dataset including only those rows whose satisfaction\_level is between 0.6 and 0.9 (inclusive).
```stan theme={null}
filter_range(ds, {"column": "satisfaction_level", "min": 0.6, "max": 0.9}) -> (ds_filtered)
```
Using the exclude parameter, the next example creates a dataset including only those rows whose satisfaction\_level falls outside the range 0.6–0.9.
```stan theme={null}
filter_range(ds, {"column": "satisfaction_level", "min": 0.6, "max": 0.9, "exclude": true}) -> (ds_filtered)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
filter_range(ds_in: dataset, {
"param": value,
...
}) -> (ds_out: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An input dataset to filter.
A new dataset containing the same columns as the input dataset but only those rows passing the filter condition.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Name of the column to apply the filter to.
If `true`, values within the specified range will be *excluded* from the resulting dataset.
Maximum value in the selected column to pass the filter (to be included).
Either this or the `min` parameter must be specified.
Minimum value in the selected column to pass the filter (to be included).
Either this or the `max` parameter must be specified.
# filter_row_numbers
Source: https://docs.graphext.com/api-docs/prepare/filter/filter_row_numbers
Filter rows by row number.
Keeps or drops rows with specific row numbers, i.e. based on their 0-based, consecutive integer index.
## Usage
The following example shows how the step can be used in a recipe.
To keep only rows with index 0, 2 and 4 (the first, third and fifth row)
```stan theme={null}
filter_row_numbers(ds, {"row_numbers": [0, 2, 4]}) -> (ds_filtered)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
filter_row_numbers(ds_in: dataset, {
"param": value,
...
}) -> (ds_out: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An input dataset to filter.
A new dataset containing the same columns as the input dataset but only those rows passing the filter condition.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Ids of the rows to filter.
Excepts either a single row (as a number) or a list of row numbers.
Each item in array.
if `true`, selected rows will be *excluded* from the resulting dataset.
# filter_rows
Source: https://docs.graphext.com/api-docs/prepare/filter/filter_rows
Filter rows using graphext's advanced query syntax (similar to Elasticsearch).
## Usage
The following examples show how the step can be used in a recipe.
This simple query creates a new dataset only including those rows where the 'age' columns is greater than 18:
```stan theme={null}
filter_rows(ds, {"query": "age: >18"}) -> (dsf)
```
Filter clients who are legally adults.
```stan theme={null}
filter_rows(ds, {"query": "age:>18"}) -> (ds_out)
```
Select clients who are exactly 19 years old.
```stan theme={null}
filter_rows(ds, {"query": "age:19"}) -> (ds_out)
```
Filter clients who are over 27 years old but below the mean age.
```stan theme={null}
filter_rows(ds, {"query": "age:>27 AND < MEAN"}) -> (ds_out)
```
Select all clients belonging to the cool class.
```stan theme={null}
filter_rows(ds, {"query": "class: cool"}) -> (ds_out)
```
Select clients belonging to the 4 most frequent classes.
```stan theme={null}
filter_rows(ds, {"query": "class: TOP(4)"}) -> (ds_out)
```
Filter clients aged 18 who earn more than 50 dollars monthly on average.
```stan theme={null}
filter_rows(ds, {"query": "(age: 18) AND ('avg monthly income':>50)"}) -> (ds_out)
```
Filter fire dates in 2020.
```stan theme={null}
filter_rows(ds, {"query": "'Fire Date': >=2020-01-01 AND <=2020-12-31"}) -> (ds_out)
```
Select rows where the text column contains both "he" and "she".
```stan theme={null}
filter_rows(ds, {"query": "text: he AND she"}) -> (ds_out)
```
Filter rows where 'Average Monthly Hours' is greater than 5 and less than 7.
```stan theme={null}
filter_rows(ds, {"query": "'Average Monthly Hours':>5 AND <7"}) -> (ds_out)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
filter_rows(ds_in: dataset, {
"param": value,
...
}) -> (ds_out: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An input dataset to filter.
A dataset containing the same columns as the input dataset but without the filtered rows.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
The *graphext advanced query* used to identify the rows to keep.
# filter_sample
Source: https://docs.graphext.com/api-docs/prepare/filter/filter_sample
Randomly sample the dataset, optionally within groups (can be used to balance a dataset).
If you request a number of rows greater than the dataframe length, it will return the original dataframe instead.
## Usage
The following examples show how the step can be used in a recipe.
This draws a sample of 12.000 random rows from the original dataset:
```stan theme={null}
filter_sample(ds, {"n_samples": 12000}) -> (ds_sampled)
```
In the next example we keep only a random half of the dataset:
```stan theme={null}
filter_sample(ds, {"n_samples": 0.5}) -> (ds_sampled)
```
And this draws a sample of 500 rows from each department identified in the original dataset (or the maximum if there are fewer than 500 rows for a particular department):
```stan theme={null}
filter_sample(ds, {"n_samples": 500, "by": "department"}) -> (ds_sampled)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
filter_sample(ds_in: dataset, {
"param": value,
...
}) -> (ds_out: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An input dataset to filter.
A new dataset containing a random sample of the original rows.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Number of rows to sample.
How many random rows to pick from the original dataset (without replacement). If the value is greater than 1,
it will be interpreted as a *count* of desired rows. If it is smaller than 1, it will be interpreted as a *proportion*
of the entire dataset.
number.
Values must be in the following range:
```javascript theme={null}
0 < {_} < 1
```
integer.
Values must be in the following range:
```javascript theme={null}
1 ≤ {_} < inf
```
Sample independently in these groups.
If a column is specified here, the sampling will be applied separately within each group defined by the unique
values in this column. Combining this with a count of rows to pick (rather than a proportion), allows this step
to balance the dataset, leading to an (approximately) equal number of rows within each group.
A value used to initialize the random number generator, making it deterministic (reproducible).
# filter_topn
Source: https://docs.graphext.com/api-docs/prepare/filter/filter_topn
Sort a dataset by selected columns and pick the first N rows (or exclude them).
## Usage
The following examples show how the step can be used in a recipe.
Keep the top 10 rows sorted by salary
```stan theme={null}
filter_topn(ds, {"n": 10, "sort_by": "salary"}) -> (ds_filtered)
```
Exclude the bottom 5 rows when sorting by date ascending
```stan theme={null}
filter_topn(ds, {"n": 5, "sort_by": "date", "ascending": true, "exclude": true}) -> (ds_filtered)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
filter_topn(ds_in: dataset, {
"param": value,
...
}) -> (ds_out: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An input dataset to filter.
A new dataset containing the same columns as the input dataset but only those rows passing the filter condition.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
How many of the leading rows to keep after sorting.
One or more columns to sort by before picking the first n rows.
May be a column name or a list of column names.
array.
Each item in array.
string.
* salary
* \['salary', 'time\_spend\_company', 'last\_evaluation']
If `true`, the first n rows after sorting will be *excluded* from the resulting dataset.
Whether to sort in ascending order rather than descending.
# filter_values
Source: https://docs.graphext.com/api-docs/prepare/filter/filter_values
Filter rows where column matches specified values exactly.
## Usage
The following examples show how the step can be used in a recipe.
To create a new dataset keeping only those rows where values in the "salary" column are either "low" or "high".
```stan theme={null}
filter_values(ds, {"column": "salary", "values": ["low", "high"]}) -> (ds_filtered)
```
Or, using the `exclude` parameter to *drop* rows where "salary" values are either "low" or "high":
```stan theme={null}
filter_values(ds, {"column": "salary", "values": ["low", "high"], "exclude": true}) -> (ds_filtered)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
filter_values(ds_in: dataset, {
"param": value,
...
}) -> (ds_out: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An input dataset to filter.
A new dataset containing the same columns as the input dataset but only those rows passing the filter condition.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Name of column to be matched against the specified `values`.
Only rows matching these values exactly will be included in the resulting dataset.
May be a single value or a list of values to be matched.
Each item in array.
* the
* \['the', 'cat']
* 2
* \[2, 3]
if `true`, only rows *not* matching the specified `values` will be included in the resulting dataset.
# filter_with_formula
Source: https://docs.graphext.com/api-docs/prepare/filter/filter_with_formula
Filter rows using a (pandas-compatible) formula.
Allowed elements in the fomula are column names as well as common operators and values for comparison
(strings need to be specified using single quotes, see example below).
For more details about valid formulas see [here](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.query.html).
## Usage
The following examples show how the step can be used in a recipe.
The first example *keeps* those rows where the "salary" column is either "low" or "high":
```stan theme={null}
filter_with_formula(ds, {
"formula": "salary == 'low' or salary == 'high'"
}) -> (ds_filtered)
```
The next example *drops* those rows where the column "number\_project" is less than 3 or greater than 4, i.e. it keeps values in the range \[3, 4] only.
```stan theme={null}
filter_with_formula(ds, {
"formula": "number_project >= 3 and number_project <= 4"
}) -> (ds_filtered)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
filter_with_formula(ds_in: dataset, {
"param": value,
...
}) -> (ds_out: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An input dataset to filter.
A new dataset containing the same columns as the input dataset but only those rows passing the filter query.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
A formula describing the matching operation to perform on each row.
* `salary == 'low' or salary == 'high'`
* `number_project >= 3 and number_project <= 4`
# Filter
Source: https://docs.graphext.com/api-docs/prepare/filter/index
| Step | Fast | Description |
| --------------------------------------------------------------------------- | ---- | ---------------------------------------------------------------------------------------- |
| [filter\_containing](/api-docs/prepare/filter/filter_containing) | | Filter rows containing any or all of a number of specified values |
| [filter\_duplicate\_nodes](/api-docs/prepare/filter/filter_duplicate_nodes) | | Remove duplicate nodes in a network |
| [filter\_duplicates](/api-docs/prepare/filter/filter_duplicates) | | Filter duplicate rows, keeping the first or last of each set of duplicates found only |
| [filter\_missing](/api-docs/prepare/filter/filter_missing) | | Filter rows based on missing values in one or more columns |
| [filter\_range](/api-docs/prepare/filter/filter_range) | | Filter rows based on the numeric values in a given column |
| [filter\_row\_numbers](/api-docs/prepare/filter/filter_row_numbers) | | Filter rows by row number |
| [filter\_rows](/api-docs/prepare/filter/filter_rows) | ⚡ | Filter rows using graphext's advanced query syntax (similar to Elasticsearch) |
| [filter\_sample](/api-docs/prepare/filter/filter_sample) | | Randomly sample the dataset, optionally within groups (can be used to balance a dataset) |
| [filter\_topn](/api-docs/prepare/filter/filter_topn) | | Sort a dataset by selected columns and pick the first N rows (or exclude them) |
| [filter\_values](/api-docs/prepare/filter/filter_values) | | Filter rows where column matches specified values exactly |
| [filter\_with\_formula](/api-docs/prepare/filter/filter_with_formula) | | Filter rows using a (pandas-compatible) formula |
| [upsample](/api-docs/prepare/filter/upsample) | | Upsample a dataset given a weight column |
# upsample
Source: https://docs.graphext.com/api-docs/prepare/filter/upsample
Upsample a dataset given a weight column.
When dealing with surveys, it's common to want your sample to reflect a specific demographic. When this ideal representation cannot be achieved, you'd usually assign a strictly positive weight to each row reflecting how representative it is of your desired population.
This step takes these precomputed weights and uses them to make the input reflect your desired population by repeating the rows a number of times in proportion to their weight until the desired image of your target population is reached within the dataset.
## Usage
The following examples show how the step can be used in a recipe.
The following example creates a new dataset with the proportions specified by `weight_name`
```stan theme={null}
upsample(ds, {"weights": "weight_name"}) -> (ds_upsampled)
```
Same as before, but ensures 3 occurences at least for the least weighted row.
```stan theme={null}
upsample(ds, {"weights": "weight_name", "n_samples_min": 3}) -> (ds_upsampled)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
upsample(ds_in: dataset, {
"param": value,
...
}) -> (ds_out: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An input dataset to upsample.
A new dataset containing the desired proportions.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Name of column to be used as weights.
Number of samples given to the least weighted set of rows.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_samples_min < inf
```
# append_rows
Source: https://docs.graphext.com/api-docs/prepare/join_and_combine/append_rows
Add rows from one dataset to another.
I.e., vertically concatenates two datasets, appending the rows of the second to the end of the first.
When the two datasets contain different columns, the `join` parameter controls whether only the
common columns are kept (`inner`), or all columns (`outer`). In the latter case, rows will have missing
values (NaNs), where a column only existed in one of the two datasets.
## Usage
The following example shows how the step can be used in a recipe.
To append the rows of dataset `ds_right` to the dataset `ds_left`, keeping all columns from both datasets:
```stan theme={null}
append_rows(ds_left, ds_right) -> (ds_out)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
append_rows(ds_left: dataset, ds_right: dataset, {
"param": value,
...
}) -> (result: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An input dataset.
A second dataset whose rows to append below the original dataset (`ds_left`).
A dataset containing the rows of both `ds_left`, and `ds_right`,
as well as an aditional column `original_index` indicating the index of each row in its original dataset.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Whether to do concatenate using an "inner" or "outer" join of columns.
When `"inner"`, only common columns will be kept. When `"outer"`, all columns will be kept.
Values must be one of the following:
* `inner`
* `outer`
# Join And Combine
Source: https://docs.graphext.com/api-docs/prepare/join_and_combine/index
| Step | Fast | Description |
| -------------------------------------------------------------- | ---- | ------------------------------------------------------------------------ |
| [append\_rows](/api-docs/prepare/join_and_combine/append_rows) | | Add rows from one dataset to another |
| [join](/api-docs/prepare/join_and_combine/join) | | Join two datasets on their row indexes or on values in specified columns |
# join
Source: https://docs.graphext.com/api-docs/prepare/join_and_combine/join
Join two datasets on their row indexes or on values in specified columns.
I.e., the equivalent of a [database join](https://en.wikipedia.org/wiki/Join_\(SQL\)) of two tables.
Adds the columns from the second dataset (`ds_right`) to the first (`ds_left`). If the two datasets contain columns
with identical names (other than those used to perform the join), configurable suffixes will be appended to their names
in the resulting dataset (see `suffixes` parameter below).
The rows included in the result depend on the kind of join (see the `how` parameter below). Depending on whether
it's a left, right, inner, or outer-join, may include rows from either dataset or both.
The join performed is always an [equi-join](https://en.wikipedia.org/wiki/Join_\(SQL\)#Equi-join), meaning that rows
from the left are matched with rows from the right where their respective values in the join column (or indexes)
are *identical* (e.g. where the value of column `id` on the left is equal to the value of column `id` on the right).
Also see [Wikipedia's article on table joins](https://en.wikipedia.org/wiki/Join_\(SQL\)) to learn more about them.
## Usage
The following example shows how the step can be used in a recipe.
For example, to enrich a dataset containing employees with information about their department:
```stan theme={null}
join(employees, departments, {
"how": "left",
"left": "department_id",
"right": "id"
}) -> (employees_dep)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
join(ds_left: dataset, ds_right: dataset, {
"param": value,
...
}) -> (result: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A dataset to join, referred to as "left" by the `how` parameter.
A second dataset to be joined with the first, referred to as "right" by the `how` parameter.
The result of the join. Contains the columns of both input datasets. Columns with identical names in both datasets
(if not used as the column joined on) will have their names concatenated with a suffix.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Type of join.
Analogous to SQL joins of the same name:
* inner: ["inner join"](https://en.wikipedia.org/wiki/Join_\(SQL\)#Inner_join). Only keeps rows where values in the
join column exist (match) in both dataset (effectively an intersection). Since it is guaranteed that each row
has values from the left and right dataset, no new missing values (NaN) will be introduced.
* left: ["left (outer) join"](https://en.wikipedia.org/wiki/Join_\(SQL\)#Left_outer_join). Keep all rows from the
*left* dataset, adding values from columns in the *right* dataset where rows match (in the join column). If no
row on the right matches, the values of columns from the right will contain NaN values only.
* right: ["right (outer) join"](https://en.wikipedia.org/wiki/Join_\(SQL\)#Right_outer_join). Keep all rows from the
*right* dataset, adding values from columns in the *left* dataset where rows match (in the join column). If no
row on the left matches, the values of columns from the left will contain NaN values only.
* outer: ["full (outer) join"](https://en.wikipedia.org/wiki/Join_\(SQL\)#Full_outer_join). Keep all rows from both
datasets. If a value in the join column doesn't match on either side, the values of columns from the non-matching
side will be NaN.
The most typical scenario is probably: "Keep my primary dataset `left` and add whatever information you can from
dataset `right`". If this is the case, you'll want a `left`-join.
Values must be one of the following:
* `left`
* `right`
* `outer`
* `inner`
Column or index in the left dataset whose values will be matched against the right.
To use the row index instead of a column use `"_index_"`, `null`, or simply omit this parameter.
Column or index in the right dataset whose values will be matched against the left.
To use the row index instead of a column use `"_index_"`, `null`, or simply omit this parameter.
Column name suffixes.
Will be appended to any original column (name) that occurs in both datasets (other than the join columns themselves).
Each item in array.
# add_noise
Source: https://docs.graphext.com/api-docs/prepare/transform/add_noise
Add noise to a column with numbers or lists of numbers.
Given a distribution name with a scale and loc parameters,
the step optionally applies another scaling to it either based on the standard deviation of the column or a proportionally to each point through
the `relative` parameter in order to preserve the underlying structure of the data. Then the computation is carried as follows:
new value = original value + relative scaling factor \* random sample from the distribution.
If this `relative` parameter is not given or is set to `abs`, then the relative scaling factor is 1.
## Usage
The following examples show how the step can be used in a recipe.
Add white noise to a column of embeddings
```stan theme={null}
add_noise(ds.embeddings) -> (ds.embeddings_with_noise)
```
Add std-dependant noise to a numerical column
```stan theme={null}
add_noise(ds.number, {"relative": "std"}) -> (ds.number_with_noise)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
add_noise(input_column: number|list[number], {
"param": value,
...
}) -> (result: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
The original column.
The result of applying noise to it.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Mode to use.
Either set to "std" to use the standard deviation, or use a number to scale the sampling.
number.
string.
Values must be one of the following:
* `std`
* `abs`
Distribution Function that noise is sampled from.
Values must be one of the following:
* `gumbel`
* `laplace`
* `logistic`
* `normal`
Mean ("centre") of the chosen distribution.
Standard deviation (spread or "width") of the distribution.
The seed to use for the random distribution, if you wish to get reproducibility in your results.
# calculate
Source: https://docs.graphext.com/api-docs/prepare/transform/calculate
Evaluates a formula containing basic arithmetic over a dataset's columns.
For example, to multiply column `A` by two and add column `Col B`, you would simply write
```java theme={null}
calculate(ds[["A", "Col B"]], {
"formula": "2 * A - `Col B`"
}) -> (ds.result)
```
For more details regarding valids operators etc. see the [Pandas eval() documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/enhancingperf.html#expression-evaluation-via-eval),
more specifically the [supported syntax](https://pandas.pydata.org/pandas-docs/stable/user_guide/enhancingperf.html#supported-syntax),
and [eval() applied to DataFrames](https://pandas.pydata.org/pandas-docs/stable/user_guide/enhancingperf.html#the-dataframe-eval-method).
Note that assignments in the formula are not supported, since the result must always be a new column,
i.e. the following kind of formula should be avoided: `"c = a + b"`. The correct way to return the result
as a column would simply be `"a + b"`.
If the name of an input column contains spaces, such as in the example above, it should be quoted in
single backticks.
## Usage
The following example shows how the step can be used in a recipe.
Assuming a dataset `ds` containing the numeric columns `num_a`, `num_b` and `num_c`, and a constant `a` with value 1.3, the following formula transforms each column numerically before adding and multiplying them together, all in one step:
```stan theme={null}
calculate(ds, {
"formula": "log(num_a) + num_b**3 * (num_c + @a)",
"constants": {"a": 1.3}
}) -> (ds.result)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
calculate(ds: dataset, {
"param": value,
...
}) -> (result: number)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A dataset with columns to be used in the evaluation of the formula. Note, all columns mentioned in the formula must be numeric!
A numeric column containing the result of evaluating the formula.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
The formula to execute.
A formula containing basic arithmetic operations and references to column names.
Constants to be used in the formula.
An object of key-value pairs, where keys refer to names of constants whose values will be available
in the formula with a `@` prefix. E.g. `"constants": {"factor": 1.23}` makes it possible to refer to
`@factor` in the formula with given value; also see example(s).
One or more constants.
Note that all constant must be of allowed types (number or string).
# cast
Source: https://docs.graphext.com/api-docs/prepare/transform/cast
Interprets and changes a column's data to another (semantic) type.
This has two consequences:
1. It will allow the resulting column to be used by steps only accepting the new type,
e.g. when casting a column of concatenated texts to the `"url"` type, so that it may be used
where Urls are expected (e.g. the step `fetch_url_content`).
2. It will change any values not conformant with the new type to the missing value (NaN). E.g.,
casting a column of mixed data containing numbers to the `"number"` type, will replace all
values that cannot be read as numbers with NaN.
Note that for each possible type a column can be cast to (via the `"type"` parameter, e.g. `"number"`,
`"category"` etc.), the steps accepts different configuration parameters. See the subsections under
[Parameters](#parameters) below for further details.
## Usage
The following examples show how the step can be used in a recipe.
E.g. to simply convert a `text` column to a `category` column, use:
```stan theme={null}
cast(ds.text, {"type": "category"}) -> (ds.new_cat)
```
To cast custom labels to the `sex` type, use `parse_labels` to map raw values to `female` and `male`:
```stan theme={null}
cast(ds.gender_text, {
"type": "sex",
"parse_labels": { "female": "woman", "male": "man" }
}) -> (ds.gender)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
cast(input: column, {
"param": value,
...
}) -> (output: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
The column you wish to cast.
A new column with original data cast to the desired type.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Desired semantic type of the converted data.
Make data numerical with `"type": "number"`.
Separator to mark the decimal part.
Use "." or "," to indicate how decimal values are separated when parsing text strings
into numerical format. It is automatically assumed that the other character is used as
the thousands separator. E.g. `"decimal": "."` assumes that the period "." is used to
separate decimals and "," thousands, as in the number string "12,173.12".
Values must be one of the following:
* `.`
* `,`
Separator to mark the thousands.
Use "." or "," to indicate how thousands are separated when parsing text strings
into numerical format. It is automatically assumed that the other character is used as
the decimal separator. E.g. `"thousand": "."` assumes that the period "." is used to
separate thousands and "," decimals, as in the number string "12.173,12".
Values must be one of the following:
* `.`
* `,`
Desired semantic type of the converted data.
Make data numerical with `"type": "list[number]"`.
A 2-character string identifying the opening and closing brackets used to identify list strings.
For example "\[]", "()", "" etc. If `null`, any possible bracket characters at the beginning and end of a
string will be removed before parsing the elements.
Separation character for split strings.
Which separation character to use to split input string into list elements.
Note that spaces will always be stripped from individual elements.
Separator to mark the decimal part.
Use "." or "," to indicate how decimal values are separated when parsing text strings
into numerical format. It is automatically assumed that the other character is used as
the thousands separator. E.g. `"decimal": "."` assumes that the period "." is used to
separate decimals and "," thousands, as in the number string "12,173.12".
Values must be one of the following:
* `.`
* `,`
Separator to mark the thousands.
Use "." or "," to indicate how thousands are separated when parsing text strings
into numerical format. It is automatically assumed that the other character is used as
the decimal separator. E.g. `"thousand": "."` assumes that the period "." is used to
separate thousands and "," decimals, as in the number string "12.173,12".
Values must be one of the following:
* `.`
* `,`
Desired semantic type of the converted data.
Make data a currency with `"type": "currency"`.
Separator to mark the decimal part.
Use "." or "," to indicate how decimal values are separated when parsing text strings
into numerical format. It is automatically assumed that the other character is used as
the thousands separator. E.g. `"decimal": "."` assumes that the period "." is used to
separate decimals and "," thousands, as in the number string "12,173.12".
Values must be one of the following:
* `.`
* `,`
Separator to mark the thousands.
Use "." or "," to indicate how thousands are separated when parsing text strings
into numerical format. It is automatically assumed that the other character is used as
the decimal separator. E.g. `"thousand": "."` assumes that the period "." is used to
separate thousands and "," decimals, as in the number string "12.173,12".
Values must be one of the following:
* `.`
* `,`
Desired semantic type of the converted data.
Make data a currency with `"type": "list[currency]"`.
A 2-character string identifying the opening and closing brackets used to identify list strings.
For example "\[]", "()", "" etc. If `null`, any possible bracket characters at the beginning and end of a
string will be removed before parsing the elements.
Separation character for split strings.
Which separation character to use to split input string into list elements.
Note that spaces will always be stripped from individual elements.
Separator to mark the decimal part.
Use "." or "," to indicate how decimal values are separated when parsing text strings
into numerical format. It is automatically assumed that the other character is used as
the thousands separator. E.g. `"decimal": "."` assumes that the period "." is used to
separate decimals and "," thousands, as in the number string "12,173.12".
Values must be one of the following:
* `.`
* `,`
Separator to mark the thousands.
Use "." or "," to indicate how thousands are separated when parsing text strings
into numerical format. It is automatically assumed that the other character is used as
the decimal separator. E.g. `"thousand": "."` assumes that the period "." is used to
separate thousands and "," decimals, as in the number string "12.173,12".
Values must be one of the following:
* `.`
* `,`
Desired semantic type of the converted data.
Convert data to the Date type with `"type": "date"`. This will allow e.g. the extraction of particular
components of the date, like year, month, or day of week (with `extract_date_components`), the calculation of
elapsed time since a given date (`time_interval`), as well as enable the use of the Trends section in graphext's
interface.
Format to parse date strings.
When input data contains strings (dates in text format), indicate how these strings are constructed.
E.g. if dates are in the format "21/07/2020", use `"format": “%d/%m/%Y”` to indicate the day, month, year order and
the use of "/" as the separator of date components. For more details on how to indicate the different
components of the date format see e.g. [Python's strftime](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes).
Unit of timestamp data.
When input data is numeric, indicates whether the numbers correspond to seconds, milliseconds, microseconds
or nanoseconds. Dates will be interpreted as so many elapsed units since the origin
(see `origin` parameter below).
For example, with `"unit": "ms"` and `"origin": "unix"` (the default), this would calculate the date
corresponding to x milliseconds since 01/01/1970, where x denotes the input numbers.
Values must be one of the following:
* `D`
* `s`
* `ms`
* `us`
* `ns`
Desired semantic type of the converted data.
Convert data to the Date type with `"type": "date"`. This will allow e.g. the extraction of particular
components of the date, like year, month, or day of week (with `extract_date_components`), the calculation of
elapsed time since a given date (`time_interval`), as well as enable the use of the Trends section in graphext's
interface.
A 2-character string identifying the opening and closing brackets used to identify list strings.
For example "\[]", "()", "" etc. If `null`, any possible bracket characters at the beginning and end of a
string will be removed before parsing the elements.
Separation character for split strings.
Which separation character to use to split input string into list elements.
Note that spaces will always be stripped from individual elements.
Format to parse date strings.
When input data contains strings (dates in text format), indicate how these strings are constructed.
E.g. if dates are in the format "21/07/2020", use `"format": “%d/%m/%Y”` to indicate the day, month, year order and
the use of "/" as the separator of date components. For more details on how to indicate the different
components of the date format see e.g. [Python's strftime](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes).
Unit of timestamp data.
When input data is numeric, indicates whether the numbers correspond to seconds, milliseconds, microseconds
or nanoseconds. Dates will be interpreted as so many elapsed units since the origin
(see `origin` parameter below).
For example, with `"unit": "ms"` and `"origin": "unix"` (the default), this would calculate the date
corresponding to x milliseconds since 01/01/1970, where x denotes the input numbers.
Values must be one of the following:
* `D`
* `s`
* `ms`
* `us`
* `ns`
Desired semantic type of the converted data.
Convert data to the Text type with `"type": "text"`. This allows the resulting column to
be used e.g. in steps involving natural language processing (NLP).
Desired semantic type of the converted data.
Convert data to the Category type with `"type": "category"`. This will influence how the
column is presented in graphext's interface, and enables the use of steps like `trim_frequencies`,
`merge_categories` etc. When converting from `list[category]`, elements will be joined using
the specified separator.
Separation character for split strings and join elements.
Which separator to use to join elements when converting from list\[category] to category.
Note that spaces will always be stripped from individual elements.
Desired semantic type of the converted data.
Convert data to the Category type with `"type": "list[category]"`. This will influence how the
column is presented in graphext's interface, and enables the use of steps like `trim_frequencies`, `merge_categories` etc.
A 2-character string identifying the opening and closing brackets used to identify list strings.
For example "\[]", "()", "" etc. If `null`, any possible bracket characters at the beginning and end of a
string will be removed before parsing the elements.
Separation character for split strings.
Which separation character to use to split input string into list elements.
Note that spaces will always be stripped from individual elements.
Desired semantic type of the converted data.
Convert data to the Url type with `"type": "url"`. This will allow e.g. fetching of any textual
content found at the specified Url (with `fetch_url_content`), or linking of a network node
in the interface to the given website (`configure_node_url`).
Desired semantic type of the converted data.
Convert data to the Url type with `"type": "list[url]"`.
A 2-character string identifying the opening and closing brackets used to identify list strings.
For example "\[]", "()", "" etc. If `null`, any possible bracket characters at the beginning and end of a
string will be removed before parsing the elements.
Separation character for split strings.
Which separation character to use to split input string into list elements.
Note that spaces will always be stripped from individual elements.
Desired semantic type of the converted data.
Convert data to the Sex type with `"type": "sex"`. This is essentially a categorical type
with two predefined values for `male` and `female`. Use `parse_labels` to configure how
raw input values should be interpreted as `female` or `male`.
Mapping of raw values to female and male categories.
An object of the form `{"female": "female_value", "male": "male_value"}` that tells the parser
which raw values in the input data should be interpreted as `female` and as `male`.
For example, `{"female": "woman", "male": "man"}` will parse `woman` as `female` and `man` as `male`.
Raw value to parse as female.
Raw value to parse as male.
Desired semantic type of the converted data.
Convert data to the Sex type with `"type": "list[sex]"`.
Use `parse_labels` to configure how raw input values are interpreted.
A 2-character string identifying the opening and closing brackets used to identify list strings.
For example "\[]", "()", "" etc. If `null`, any possible bracket characters at the beginning and end of a
string will be removed before parsing the elements.
Separation character for split strings.
Which separation character to use to split input string into list elements.
Note that spaces will always be stripped from individual elements.
Mapping of raw values to female and male categories.
An object of the form `{"female": "female_value", "male": "male_value"}` that tells the parser
which raw values in the input data should be interpreted as `female` and as `male`.
For example, `{"female": "woman", "male": "man"}` will parse `woman` as `female` and `man` as `male`.
Raw value to parse as female.
Raw value to parse as male.
Desired semantic type of the converted data.
Convert data to the Boolean (logical) type with `"type": "boolean"`. If the input data is numeric,
0s will be treated as False and all other values as True. If the input data contains text strings,
the values in lower- or uppercase will be interpreted as True, and the
values as False. Any remaining values will be converted to NaN (missing).
Desired semantic type of the converted data.
Convert data to the Boolean (logical) type with `"type": "list[boolean]"`. If the input data is numeric, 0s will be treated as False and all other values as True. If the input data contains text strings, the values in lower- or uppercase will be interpreted as True, and the values as False. Any remaining values will be converted to NaN (missing).
A 2-character string identifying the opening and closing brackets used to identify list strings.
For example "\[]", "()", "" etc. If `null`, any possible bracket characters at the beginning and end of a
string will be removed before parsing the elements.
Separation character for split strings.
Which separation character to use to split input string into list elements.
Note that spaces will always be stripped from individual elements.
# concatenate
Source: https://docs.graphext.com/api-docs/prepare/transform/concatenate
Concatenate columns as text or lists with optional separator as well as pre- and postfix.
If only a single input column is provided, even if it is a list, the result will be a text column by default.
If multiple columns are passed, and any of these contains lists, then the result is also a column of lists.
In this case, each output list will contain the result of concatenating all elements in the corresponding row,
whether these elements are themselves lists or not.
If none of the multiple input columns contains lists, the result will be a text column. Each input column will be converted
to a string representation if necessary, and then concatenated with a given separator and pre- and/or postfix.
You can change this default behavior by explicitly setting an `out_type` in params.
## Usage
The following examples show how the step can be used in a recipe.
The following example combines first names, last names and a title to create a new column with values in the form "Dr. first\_name last\_name":
```stan theme={null}
concatenate(ds.first_name, ds.last_name, {
"separator": " ",
"prefix": "Dr. "
}) -> (ds.title_fullname)
```
Another example simply prefixes domain names of the form "twitter.com" or "google.com" to create URLs of the form "[https://www.twitter.com](https://www.twitter.com)" etc...
```stan theme={null}
concatenate(ds.domain, {
"separator": null,
"prefix": "https://www."
}) -> (ds.full_url)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
concatenate(*columns: column, {
"param": value,
...
}) -> (result: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
One or more columns to concatenate.
Column containing the result of the concatenation.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
A separator to use between elements of individual columns when concatenating as texts.
A prefix to prepend to the result of the concatenation (or to a single column if no more were provided).
A postfix to append to the result of the concatenation (or to a single column if no more were provided).
How to represent missing values (NaN) in the concatenated result.
If a "nan\_as" value is specified, this will be used to fill in missing values during concatenation.
With `"nan_as": null` the concatenation will produce a missing value in rows where at least 1 column
to be concatenated had a missing value.
The semantic data type of the output column.
Note, if this type is not compatible with the result of the concatenation, the output may consist of missing values
(NaNs) only.
Values must be one of the following:
`category` `date` `number` `currency` `url` `boolean` `text` `list[category]` `list[date]` `list[number]` `list[currency]` `list[url]` `list[boolean]`
# count_unique
Source: https://docs.graphext.com/api-docs/prepare/transform/count_unique
Counts the number of unique elements in each list/array of the input column.
## Usage
The following example shows how the step can be used in a recipe.
This step has no configuration parameters, so it's simply:
```stan theme={null}
count_unique(ds.input_lists) -> (ds.n_unique)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
count_unique(input: list) -> (output: number)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column containing lists.
The count of unique elements for each input list.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
This step doesn't expect any configuration.
# derive_column
Source: https://docs.graphext.com/api-docs/prepare/transform/derive_column
Derive a new column with a custom JS script.
Supports any JS script using ECMAScript 2023 syntax. The script should have a `return` clause returning either a value or null / undefined.
The script has access to a `row` object that represent a row in the dataset and have the column names as keys.
Lists are supported both as inputs and outputs.
It's important to correctly manage null values by checking for null (e.g. `if (row.col != null) { ... }`) or using the JS optional chaining operator (`?`).
## Usage
The following examples show how the step can be used in a recipe.
The following example joins all values in a list of numbers with '|' as separator:
```stan theme={null}
derive_column(ds, {
"script": "return row.numCol?.join(' | ');",
"type": "text"
}) -> (ds.new_col)
```
The following example computes the sum for a list of numbers:
```stan theme={null}
derive_column(ds, {
"script": "return row.numCol?.reduce((sum, n) => sum + n, 0);",
"type": "category"
}) -> (ds.new_col)
```
The following example adds a prefix to a category:
```stan theme={null}
derive_column(ds, {
"script": "return row.cat != null ? `Prefix_${row.cat}` : null;"
}) -> (ds.new_col)
```
The following example extracts a regex from a text:
```stan theme={null}
derive_column(ds, {
"script": "return row.text?.match(/\d+/);",
"type": "category"
}) -> (ds.new_col)
```
The following example extracts the domain from a URL column:
```stan theme={null}
derive_column(ds, {
"script": "return row.url != null ? new URL(row.url).hostname : null;",
"type": "category"
}) -> (ds.new_col)
```
The following example extracts the year component from a Date column:
```stan theme={null}
derive_column(ds, {
"script": "return row.dateCol != null ? new Date(row.dateCol).getUTCFullYear() : null;",
"type": "number"
}) -> (ds.new_col)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
derive_column(ds: dataset, {
"param": value,
...
}) -> (new_col: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An input dataset.
The column resulting from evaluating the script.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
The javascript code to execute.
* For example, to multiply by 2 every row with a value:
```json theme={null}
"return row.num * 2;"
```
Output column type.
Select the desired type using a shortened yet fully specified name.
Values must be one of the following:
`boolean` `category` `date` `number` `text` `url` `list[number]` `list[category]` `list[url]` `list[date]` `list[boolean]`
# discretize_on_quantiles
Source: https://docs.graphext.com/api-docs/prepare/transform/discretize_on_quantiles
Discretize column into bins based on quantiles.
Quantiles can be defined as an array of cut points (e.g., \[0.25, 0.5, 0.75]) or as a number indicating the desired number of bins.
Each bin can optionally be assigned a label.
## Usage
The following examples show how the step can be used in a recipe.
The following parameters will discretize a column with values in the range \[0, 1], producing a new categorical column with bins defined by the borders \[0, 0.25), \[0.25, 0.5), \[0.5, 0.75) and \[0.75, 1]. The bins will have labels "q1", "q2", "q3" and "q4" respectively.
```stan theme={null}
discretize_on_quantiles(ds.price, {
"quantiles": [0.25, 0.5, 0.75],
"labels": ["q1", "q2", "q3", "q4"]
}) -> (ds.price_category)
```
The following parameters will discretize a column into 4 equally sized bins.
```stan theme={null}
discretize_on_quantiles(ds.age, {
"quantiles": 4
}) -> (ds.age_category)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
discretize_on_quantiles(input: number, {
"param": value,
...
}) -> (output: category)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A numeric column to discretize.
A new categorical column with categories corresponding to discretized bins.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Quantiles.
Defines the quantiles or number of bins for discretizing the column.
Can be an array of cut points or a number indicating the number of bins.
Values must be in the following range:
```javascript theme={null}
2 ≤ quantiles < inf
```
Each item in array.
Values must be in the following range:
```javascript theme={null}
0 ≤ Item ≤ 1
```
* 4
* \[0.25, 0.5, 0.75]
Names for the resulting bins.
Needs one more label than the number of quantile cut points.
Each item in array.
* \['Q1', 'Q2', 'Q3', 'Q4']
# discretize_on_values
Source: https://docs.graphext.com/api-docs/prepare/transform/discretize_on_values
Discretize column by binning its values using explicitly specified cuts points.
Each bin can optionally be assigned a label.
## Usage
The following example shows how the step can be used in a recipe.
The following parameters will discretize a column with values in the range \[0, 1], producing a new categorical column with bins (0, 0.33], (0.33, 0.66], (0.66, 1] (i.e. right-inclusive). The bins will be labelled "low", "medium", and "high" respectively.
```stan theme={null}
discretize_on_values(ds.price, {"cuts": [0.33, 0.66], "add_extremes": true, "labels": ["low", "medium", "high"]}) -> (ds.price_category)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
discretize_on_values(input: number, {
"param": value,
...
}) -> (output: category)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A quantitative column to discretize.
A new categorical column with categories corresponding to discretized bins.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Points/values used to cut the quantitative column into bins.
Each item in array.
Names for the resulting bins.
*Important: Note that cutting a series of values in 3 places creates 4 bins.*.
Each item in array.
Whether to automatically include the minimum and maximum values of the column as cut points.
Whether the intervals are right-inclusive, i.e. of the form `(x1, x2]`, or left-inclusive `[x1, x2)`.
# divide
Source: https://docs.graphext.com/api-docs/prepare/transform/divide
Divide two or more numeric columns in given order.
An additional constant may be used to divide the final result by.
## Usage
The following examples show how the step can be used in a recipe.
To simply divide one column by another:
```stan theme={null}
divide(ds.numerator, ds.denominator) -> (ds.quotient)
```
To divide a single column by 2.0:
```stan theme={null}
divide(ds.num_column, {"constant": 2.0}) -> (ds.result)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
divide(*columns: number, {
"param": value,
...
}) -> (result: number)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
One or more numeric columns to divide (left to right).
The result of the division.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
A constant to divide by.
Whether to ignore NaNs or propagate them.
# equal
Source: https://docs.graphext.com/api-docs/prepare/transform/equal
Check the row-wise equality of all input columns.
For each row, checks whether all values in that row are equal. The result is a boolean column
indicating equality for each row as `true` or `false`.
Note that if the types of input columns are not compatible, the result will be `False` for all
rows. Compatibility here means that input columns must be
* all numeric or boolean (the latter being interpreted as 0.0/1.0), OR
* all string-like (categorical or text), OR
* all list-like
By default, missing values (NaNs) in the same location are considered equal in this step. However,
check the parameter `keep_nans` below to control how the presence of NaNs affects the result.
Also, when performing numeric comparison, the parameters `rel_tol` and `abs_tol` can be used to check
for approximate equality. The desired tolerance (precision) can then be expressed either as a
proportion of a reference value; and/or as an absolute maximum difference). More specifically,
the equation used to check for numeric equality between values `a` and `b` is:
`absolute(a - b) <= (rel_tol * absolute(b) + abs_tol)`.
Also see the parameter descriptions below, or the corresponding
[numpy documentation](https://numpy.org/doc/stable/reference/generated/numpy.isclose.html)
for further details.
## Usage
The following examples show how the step can be used in a recipe.
To check exact equality of numeric columns `num1` and `num2`
```stan theme={null}
equal(ds.num1, ds.num2) -> (ds.num1_num2_eq)
```
To check *approximate* equality of numeric columns `num1` and `num2`, with differences of less than 0.001 being considered "equal" use the `abs_tol` parameter, (note that for reasons of limited precision in how numbers are stored it would be safer to use e.g. 0.0011 or even 0.002 to approximate equality to three decimals):
```stan theme={null}
equal(ds.num1, ds.num2, {"abs_tol": 0.001}) -> (ds.aprox_eq)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
equal(*columns: column, {
"param": value,
...
}) -> (result: boolean)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
One or more columns to check for equality.
Output column indicating row-wise equality of the input columns.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Absolute tolerance.
The absolute (positive) difference of two numbers must be smaller than or equal to this value
for them to be considered equal.
Relative tolerance.
The absolute (positive) difference of two numbers `a` and `b` must be smaller than or equal
to `rel_tol * absolute(b)` for them to be considered equal.
Whether to maintain missing values (NaNs) in the result.
The possible values are `{true, false, "any", "all"}`:
* If `false`: use default NaN comparison. I.e. `NaN == value => false` but `NaN == NaN => true`.
Note that this means the result will never contain any NaNs.
* If `true` or `any`: the result will be NaN if *any* value in a row is NaN
* If `all`: the result will be NaN if *all* values in a row are NaN.
Values must be one of the following:
* `any`
* `all`
* `True`
* `False`
# explode
Source: https://docs.graphext.com/api-docs/prepare/transform/explode
Explode (extract) items from column(s) of lists into separate rows.
Each element from an exploded list will results in a new *row* in the resulting dataset, i.e. the tranformation will create a *taller* dataset than the original, but one that has the same number of columns.
Note: to unpack lists into separate *columns*, see the step `unpack_list` instead.
## Usage
The following examples show how the step can be used in a recipe.
Explode all list columns into separate rows
```stan theme={null}
explode(ds) -> (ds_exploded)
```
Explode only the tags column, keeping specific columns
```stan theme={null}
explode(ds, {"explode_by": "tags", "just_keep": ["title", "tags"]}) -> (ds_exploded)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
explode(ds: dataset, {
"param": value,
...
}) -> (ds_out: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An input dataset having at least one column containing lists.
A taller output dataset having *no* list columns.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
The list of columns to explode.
Any list columns in the input dataset not mentioned here will not be exploded, i.e. will
remain list columns in the output dataset. If `null`, attempts to explode *all* columns
containing lists.
null.
string.
array.
Each item in array.
Columns to keep in the output dataset.
Specifies which non-exploded columns should be included in the output dataset. If `null` (default),
all non-exploded columns will be included. If a string, only that column will be included. If an array
of strings, only those columns will be included. Note that columns specified in `explode_by` will always
be included regardless of this parameter.
null.
string.
array.
Each item in array.
Whether to explode the selected columns together.
If `true`, assumes all specified columns to be exploded are of the same lengths (in any
given row). In this case, if a row contains two lists with 5 elements each, this will
produce *5* rows with matching elements in the output dataset.
If `false`, on the other hand, will explode iteratively column-by-column. A row containing
two lists with 5 elements each, will therefore produce *25* rows in the output dataset. I.e.,
exploding the first column will produce 5 rows, and when these rows are exploded again
using the second column, each will produce 5 rows in turn.
The *graphext advanced query* used to identify the rows to select previous to the grouping.
# extract_date_component
Source: https://docs.graphext.com/api-docs/prepare/transform/extract_date_component
Extract a component such as day, week, weekday etc. from a date column.
The type of the output column depends on the component extracted. Where the component is a name (e.g. of the day of week),
the result will be a categorical column. Otherwise it will be numeric.
## Usage
The following example shows how the step can be used in a recipe.
To extract the number of the month (1..12) from a date column:
```stan theme={null}
extract_date_component(ds.date, {"component": "month"}) -> (ds.month)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
extract_date_component(date: date, {
"param": value,
...
}) -> (component: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column of dates to extract the desired component from.
A numeric or categorical column containing the desired component.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
The name of the date component to extract.
Values must be one of the following:
`date` `day` `day_name` `dayofmonth` `dayofweek` `dayofyear` `hour` `minute` `month` `month_name` `part_of_day` `partofday` `period_of_day` `periodofday` `quarter` `season` `second` `time` `week` `weekday` `weekday_name` `weekofyear` `year`
The timezone to use when extracting the date component.
The timezone to use when extracting the date component. If not specified, the timezone of the column metadata will be used.
If the column metadata does not specify a timezone, UTC will be used.
The expected format is \[+/-]HH:MM, e.g. +01:00, 02:00, -05:00, +00:00, etc.
# extract_emoji
Source: https://docs.graphext.com/api-docs/prepare/transform/extract_emoji
Parse texts and extract their emoji.
Generates a new column with one list of emoji for each original text (row).
## Usage
The following example shows how the step can be used in a recipe.
Without configuring the languages to be processed simply use the following code. Otherwise see parameters below.
```stan theme={null}
extract_emoji(ds.text, ds.lang) -> (ds.emoji)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
extract_emoji(text: text, *lang: category, {
"param": value,
...
}) -> (emoji: list[category])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A text column to extract emoji from.
An (optional) column identifying the languages of the corresponding texts. It is used to identify the correct model (spaCy)
to use for each text. If the dataset doesn't contain such a column yet, it can be created using the `infer_language` step.
Ideally, languages should be expressed as two-letter
[ISO 639-1 language codes](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes), such as "en", "es" or "de" for
English, Spanish or German respectively. We also detect fully spelled out names such as "english", "German", "allemande"
etc., but it is not guaranteed that we will recognize all possible spellings correctly always, so ISO codes should be
preferred.
Alternatively, if all texts are in the same language, it can be identified with the `language` *parameter* instead.
A column of emojis extracted from each text.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Whether to enable support for additional languages.
By default, Arabic ("ar"), Catalan ("ca"), Basque ("eu"), and Turkish ("tu") are not enabled,
since they're supported only by a different class of language models (stanfordNLP's Stanza)
that is much slower than the rest. This parameter can be used to enable them.
Minimum number (or proportion) of texts to include a language in processing.
Any texts in a language with fewer documents than these will be ignored. Can be useful to speed up
processing when there is noise in the input languages, and when ignoring languages with a small number of
documents only is acceptable. Values smaller than 1 will be interpreted as a *proportion* of all texts, and
values greater than or equal to 1 as an *absolute number* of documents.
number.
Values must be in the following range:
```javascript theme={null}
0 < {_} < 1
```
integer.
Values must be in the following range:
```javascript theme={null}
1 ≤ {_} < inf
```
The language of inputs texts.
If all texts are in the same language, it can be specified here instead of passing it as an input column. The language will be used to identify the correct spaCy model to parse and analyze the texts. For allowed values, see the comment regarding the `lang` column above.
# extract_entities
Source: https://docs.graphext.com/api-docs/prepare/transform/extract_entities
Parse texts and extract the entities mentioned (persons, organizations etc.).
Generates one column per type of entity (see below), each containing lists of entities detected in the corresponding text.
## Usage
The following example shows how the step can be used in a recipe.
To extract entities for all languages supported by default simply use the following code. Otherwise see parameters below.
```stan theme={null}
extract_entities(ds.text, ds.lang) -> (ds.emoji)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
extract_entities(text: text, *lang: category, {
"param": value,
...
}) -> (
People: list[category],
Groups: list[category],
Organizations: list[category],
GPEs: list[category],
Locations: list[category],
Products: list[category],
Events: list[category],
Money: list[category]
)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A text column to extract entities from.
An (optional) column identifying the languages of the corresponding texts. It is used to identify the correct model (spaCy)
to use for each text. If the dataset doesn't contain such a column yet, it can be created using the `infer_language` step.
Ideally, languages should be expressed as two-letter
[ISO 639-1 language codes](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes), such as "en", "es" or "de" for
English, Spanish or German respectively. We also detect fully spelled out names such as "english", "German", "allemande"
etc., but it is not guaranteed that we will recognize all possible spellings correctly always, so ISO codes should be
preferred.
Alternatively, if all texts are in the same language, it can be identified with the `language` *parameter* instead.
Lists of people detected in the texts (including fictional).
Lists of nationalities and religious or political groups detected in the texts.
Lists of organizations detected in the texts (companies, agencies, institutions, etc.).
Lists of geo-political entities detected in the texts, i.e. countries, cities, states etc.
Lists of locations detected in the texts, other than GPEs, such as mountain ranges, bodies of water etc.
Lists of products detected in the texts (objects, vehicles, foods, etc., not services).
Lists of events detected in the texts (e.g. named hurricanes, battles, wars, sports events, etc.).
List of monetary values detected in the texts, including unit.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Whether to enable support for additional languages.
By default, Arabic ("ar"), Catalan ("ca"), Basque ("eu"), and Turkish ("tu") are not enabled,
since they're supported only by a different class of language models (stanfordNLP's Stanza)
that is much slower than the rest. This parameter can be used to enable them.
Minimum number (or proportion) of texts to include a language in processing.
Any texts in a language with fewer documents than these will be ignored. Can be useful to speed up
processing when there is noise in the input languages, and when ignoring languages with a small number of
documents only is acceptable. Values smaller than 1 will be interpreted as a *proportion* of all texts, and
values greater than or equal to 1 as an *absolute number* of documents.
number.
Values must be in the following range:
```javascript theme={null}
0 < {_} < 1
```
integer.
Values must be in the following range:
```javascript theme={null}
1 ≤ {_} < inf
```
The language of inputs texts.
If all texts are in the same language, it can be specified here instead of passing it as an input column. The language will be used to identify the correct spaCy model to parse and analyze the texts. For allowed values, see the comment regarding the `lang` column above.
# extract_hashtags
Source: https://docs.graphext.com/api-docs/prepare/transform/extract_hashtags
Parse texts and extract any hashtags mentioned.
Generates a new column with one list of hashtags (words starting with the symbol "#") for each original text.
## Usage
The following example shows how the step can be used in a recipe.
To extract hashtags for all languages supported by default simply use the following code. Otherwise see parameters below.
```stan theme={null}
extract_hashtags(ds.text, ds.lang) -> (ds.hashtags)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
extract_hashtags(text: text, *lang: category, {
"param": value,
...
}) -> (hashtags: list[category])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A text column to extract hashtags from.
An (optional) column identifying the languages of the corresponding texts. It is used to identify the correct model (spaCy)
to use for each text. If the dataset doesn't contain such a column yet, it can be created using the `infer_language` step.
Ideally, languages should be expressed as two-letter
[ISO 639-1 language codes](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes), such as "en", "es" or "de" for
English, Spanish or German respectively. We also detect fully spelled out names such as "english", "German", "allemande"
etc., but it is not guaranteed that we will recognize all possible spellings correctly always, so ISO codes should be
preferred.
Alternatively, if all texts are in the same language, it can be identified with the `language` *parameter* instead.
A column of lists containing the hashtags extracted from the texts.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Whether to enable support for additional languages.
By default, Arabic ("ar"), Catalan ("ca"), Basque ("eu"), and Turkish ("tu") are not enabled,
since they're supported only by a different class of language models (stanfordNLP's Stanza)
that is much slower than the rest. This parameter can be used to enable them.
Minimum number (or proportion) of texts to include a language in processing.
Any texts in a language with fewer documents than these will be ignored. Can be useful to speed up
processing when there is noise in the input languages, and when ignoring languages with a small number of
documents only is acceptable. Values smaller than 1 will be interpreted as a *proportion* of all texts, and
values greater than or equal to 1 as an *absolute number* of documents.
number.
Values must be in the following range:
```javascript theme={null}
0 < {_} < 1
```
integer.
Values must be in the following range:
```javascript theme={null}
1 ≤ {_} < inf
```
The language of inputs texts.
If all texts are in the same language, it can be specified here instead of passing it as an input column. The language will be used to identify the correct spaCy model to parse and analyze the texts. For allowed values, see the comment regarding the `lang` column above.
# extract_json_values
Source: https://docs.graphext.com/api-docs/prepare/transform/extract_json_values
Extract values from JSON columns using JsonPath.
Uses a simplified JsonPath-like syntax to extract values from JSON objects.
**Supported syntax:**
* Dot notation: `address.city`, `address.anotherLevel.key`
* Array index: `phoneNumbers[0].type`, `phoneNumbers[1].number`
* Array slice (all elements): `phoneNumbers[:]`, `phoneNumbers[::]`
* Array slice (range): `phoneNumbers[0:2]`, `phoneNumbers[0:2:1]`
* Array slice (with step): `phoneNumbers[::2]`
* Quoted keys (for special characters): `address["other info"]`
* Root array access: `$[:].firstName`
**Not supported:**
* Wildcard `[*]` — use `[:]` instead
* Negative indices `[-1]`
* Recursive descent `..`
See [darro#243](https://github.com/graphext/darro/issues/243) for tracking support of these operators.
## Usage
The following examples show how the step can be used in a recipe.
Extract a simple nested value.
```stan theme={null}
extract_json_values(ds.json_col, {
"path": "address.city",
"type": "text"
}) -> (ds.cities)
```
Extract a value from the first element of an array.
```stan theme={null}
extract_json_values(ds.json_col, {
"path": "phoneNumbers[0].type",
"type": "category"
}) -> (ds.first_phone_type)
```
Extract values from all elements of an array using slice notation.
```stan theme={null}
extract_json_values(ds.json_col, {
"path": "phoneNumbers[:].type",
"type": "list[category]"
}) -> (ds.all_phone_types)
```
Extract values from a range of array elements.
```stan theme={null}
extract_json_values(ds.json_col, {
"path": "items[0:3].name",
"type": "list[category]"
}) -> (ds.first_three_names)
```
Extract a deeply nested value.
```stan theme={null}
extract_json_values(ds.json_col, {
"path": "address.anotherLevel.key",
"type": "text"
}) -> (ds.deep_value)
```
Access a key with special characters using quoted bracket notation.
```stan theme={null}
extract_json_values(ds.json_col, {
"path": "address[\"other info\"]",
"type": "text"
}) -> (ds.other_info)
```
Extract from root-level arrays using the \$ symbol.
```stan theme={null}
extract_json_values(ds.json_col, {
"path": "$[:].firstName",
"type": "list[category]"
}) -> (ds.first_names)
```
Extract numeric values from an array as a multivalued number column.
```stan theme={null}
extract_json_values(ds.json_col, {
"path": "scores",
"type": "list[number]"
}) -> (ds.all_scores)
```
Extract dates in ISO 8601 format (e.g. "2024-01-15T10:30:00" or "2024-01-15"). These are parsed automatically.
```stan theme={null}
extract_json_values(ds.json_col, {
"path": "created_at",
"type": "date"
}) -> (ds.created_at)
```
For dates in non-standard formats (e.g. "15/01/2024", "01-15-2024"), extract as category first and then use cast with a format string, because extract\_json\_values does not support custom date format parsing.
```stan theme={null}
extract_json_values(ds.json_col, {
"path": "event_date",
"type": "category"
}) -> (ds.event_date_raw)
cast(ds.event_date_raw, {
"type": "date",
"format": "%d/%m/%Y"
}) -> (ds.event_date)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
extract_json_values(text: text|category, {
"param": value,
...
}) -> (value_extracted: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A text column with Json values to extract parts from.
The column resulting from evaluating the JsonPath expression on the input column.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
JsonPath-like string used to extract values from the JSON column. Supports dot notation, array indices, slices and quoted keys. Does not support wildcard \[\*], negative indices or recursive descent (..).
* address.city
* phoneNumbers\[:].type
Output column type.
Select the desired type using a shortened yet fully specified name.
Values must be one of the following:
`boolean` `category` `date` `number` `text` `url` `list[number]` `list[category]` `list[url]`
# extract_keywords
Source: https://docs.graphext.com/api-docs/prepare/transform/extract_keywords
Parse and extract keywords from texts.
The text elements considered keywords are configurable. They can include detected noun phrases (compound nouns
like 'the quick brown fox'), any automatically recognized entities (people, products, events), or any lexical
category of word, such as nouns, verbs, adjectives etc.
## Usage
The following examples show how the step can be used in a recipe.
To extract all kinds of nouns only, i.e. entities, compound nouns, simple nouns and proper nouns (names):
```stan theme={null}
extract_keywords(ds.text, ds.lang,
{
"keywords": {
"entities": true,
"noun_phrases": true,
"pos_tags": ["NOUN", "PROPN"]
}
}) -> (ds.keywords)
```
To also include adjectives, and limit keywords to those that occur in at least 3 but no more than 90% of all documents:
```stan theme={null}
extract_keywords(ds.text, ds.lang,
{
"keywords": {
"entities": true,
"noun_phrases": true,
"pos_tags": ["NOUN", "PROPN", "ADJ"],
"frequency_filter": {
"min_rows": 3,
"max_rows": 0.9
}
}
}) -> (ds.keywords)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
extract_keywords(text: text, *lang: category, {
"param": value,
...
}) -> (keywords: list[category])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A text column to extract keywords from.
An (optional) column identifying the languages of the corresponding texts. It is used to identify the correct model (spaCy)
to use for each text. If the dataset doesn't contain such a column yet, it can be created using the `infer_language` step.
Ideally, languages should be expressed as two-letter
[ISO 639-1 language codes](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes), such as "en", "es" or "de" for
English, Spanish or German respectively. We also detect fully spelled out names such as "english", "German", "allemande"
etc., but it is not guaranteed that we will recognize all possible spellings correctly always, so ISO codes should be
preferred.
Alternatively, if all texts are in the same language, it can be identified with the `language` *parameter* instead.
Lists containing the keywords mentioned in each text.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Whether to enable support for additional languages.
By default, Arabic ("ar"), Catalan ("ca"), Basque ("eu"), and Turkish ("tu") are not enabled,
since they're supported only by a different class of language models (stanfordNLP's Stanza)
that is much slower than the rest. This parameter can be used to enable them.
Minimum number (or proportion) of texts to include a language in processing.
Any texts in a language with fewer documents than these will be ignored. Can be useful to speed up
processing when there is noise in the input languages, and when ignoring languages with a small number of
documents only is acceptable. Values smaller than 1 will be interpreted as a *proportion* of all texts, and
values greater than or equal to 1 as an *absolute number* of documents.
number.
Values must be in the following range:
```javascript theme={null}
0 < {_} < 1
```
integer.
Values must be in the following range:
```javascript theme={null}
1 ≤ {_} < inf
```
The language of inputs texts.
If all texts are in the same language, it can be specified here instead of passing it as an input column. The language will be used to identify the correct spaCy model to parse and analyze the texts. For allowed values, see the comment regarding the `lang` column above.
Configure how keywords are extracted.
Define the text elements considered keywords and their minimum or maximum frequency in the dataset to be
included in the result.
Part-Of-Speech (POS) tags.
Which lexical units (nouns, verbs etc.) to include as keywords.
See [spaCy's universal part-of-speech tags](https://spacy.io/api/annotation#section-pos-tagging)
for a detailed table of allowed values.
Each item in array.
Values must be one of the following:
`ADJ` `ADP` `ADV` `AUX` `CONJ` `CCONJ` `DET` `INTJ` `NOUN` `NUM` `PART` `PRON` `PROPN` `PUNCT` `SCONJ` `SYM` `VERB`
Whether or not to include any detected entities (people, places, events, etc.).
Whether or not to include compound noun phrases (such as 'the quick red fox').
Filter keywords based on the number of texts they occur in.
Filter conditions can be applied globally or per language.
Minimum number of rows.
Keywords not occurring in at least these many rows (texts) will be excluded.
Values must be in the following range:
```javascript theme={null}
0 ≤ min_rows < inf
```
Maximum proportion of rows.
Keywords occurring in more than this proportion of rows (texts) will be excluded.
Values must be in the following range:
```javascript theme={null}
0 ≤ max_rows ≤ 1
```
Whether to always include the n most frequent keywords.
I.e. independent of any other filter conditions. Set to `null` to ignore.
Values must be in the following range:
```javascript theme={null}
0 ≤ keep_top_n < inf
```
Whether to exclude n most frequent keywords.
I.e. independent of any other filter conditions.
Values must be in the following range:
```javascript theme={null}
0 ≤ filter_top_n < inf
```
Filter per language.
Apply filter conditions separately to texts grouped by language, rather than across all texts.
# extract_mentions
Source: https://docs.graphext.com/api-docs/prepare/transform/extract_mentions
Parse texts and extract any mentions detected.
A "mention", i.e. a reference to a user or account, here simply means any word starting with the "@" character.
The step generates a new column with one list of mentions for each original text (row).
## Usage
The following example shows how the step can be used in a recipe.
Without configuring the languages to be processed simply use the following code. Otherwise see parameters below.
```stan theme={null}
extract_mentions(ds.text, ds.lang) -> (ds.mentions)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
extract_mentions(text: text, *lang: category, {
"param": value,
...
}) -> (mentions: list[category])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A text column to extract mentions from.
An (optional) column identifying the languages of the corresponding texts. It is used to identify the correct model (spaCy)
to use for each text. If the dataset doesn't contain such a column yet, it can be created using the `infer_language` step.
Ideally, languages should be expressed as two-letter
[ISO 639-1 language codes](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes), such as "en", "es" or "de" for
English, Spanish or German respectively. We also detect fully spelled out names such as "english", "German", "allemande"
etc., but it is not guaranteed that we will recognize all possible spellings correctly always, so ISO codes should be
preferred.
Alternatively, if all texts are in the same language, it can be identified with the `language` *parameter* instead.
A column of lists containing the mentions extracted from the texts.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Whether to enable support for additional languages.
By default, Arabic ("ar"), Catalan ("ca"), Basque ("eu"), and Turkish ("tu") are not enabled,
since they're supported only by a different class of language models (stanfordNLP's Stanza)
that is much slower than the rest. This parameter can be used to enable them.
Minimum number (or proportion) of texts to include a language in processing.
Any texts in a language with fewer documents than these will be ignored. Can be useful to speed up
processing when there is noise in the input languages, and when ignoring languages with a small number of
documents only is acceptable. Values smaller than 1 will be interpreted as a *proportion* of all texts, and
values greater than or equal to 1 as an *absolute number* of documents.
number.
Values must be in the following range:
```javascript theme={null}
0 < {_} < 1
```
integer.
Values must be in the following range:
```javascript theme={null}
1 ≤ {_} < inf
```
The language of inputs texts.
If all texts are in the same language, it can be specified here instead of passing it as an input column. The language will be used to identify the correct spaCy model to parse and analyze the texts. For allowed values, see the comment regarding the `lang` column above.
# extract_ngrams
Source: https://docs.graphext.com/api-docs/prepare/transform/extract_ngrams
Parse texts and extract their n-grams.
An [n-gram](https://en.wikipedia.org/wiki/N-gram) here means a contiguous sequence of n words in the original text.
The step extracts all n-grams of a given text, i.e. starting at each individual word in original order. The result
is one list of n-grams per input text, where each n-gram is a single text string with individual words separated by
spaces (unless configured otherwise). The maximum size and kind of n-grams extracted, as well as how to represent them
in the result can be configured via the parameters described below. The step also allows filtering of n-grams based
on their frequency in the dataset.
## Usage
The following example shows how the step can be used in a recipe.
Using a custom configuration to select the maximum size (*n*) of the n-grams, and how to represent them:
```stan theme={null}
extract_ngrams(ds.text, ds.lang, {
"ngrams": {
"n_max": 4,
"filters": ["punct", "stops"],
"attrib": "lower",
"unigram_lemmas": true,
"concat": false
}
}) -> (ds.ngrams)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
extract_ngrams(text: text, *lang: category, {
"param": value,
...
}) -> (ngrams: list[category])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A text column to extract n-grams from.
An (optional) column identifying the languages of the corresponding texts. It is used to identify the correct model (spaCy)
to use for each text. If the dataset doesn't contain such a column yet, it can be created using the `infer_language` step.
Ideally, languages should be expressed as two-letter
[ISO 639-1 language codes](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes), such as "en", "es" or "de" for
English, Spanish or German respectively. We also detect fully spelled out names such as "english", "German", "allemande"
etc., but it is not guaranteed that we will recognize all possible spellings correctly always, so ISO codes should be
preferred.
Alternatively, if all texts are in the same language, it can be identified with the `language` *parameter* instead.
A column of lists containing the n-grams extracted from the texts.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Whether to enable support for additional languages.
By default, Arabic ("ar"), Catalan ("ca"), Basque ("eu"), and Turkish ("tu") are not enabled,
since they're supported only by a different class of language models (stanfordNLP's Stanza)
that is much slower than the rest. This parameter can be used to enable them.
Minimum number (or proportion) of texts to include a language in processing.
Any texts in a language with fewer documents than these will be ignored. Can be useful to speed up
processing when there is noise in the input languages, and when ignoring languages with a small number of
documents only is acceptable. Values smaller than 1 will be interpreted as a *proportion* of all texts, and
values greater than or equal to 1 as an *absolute number* of documents.
number.
Values must be in the following range:
```javascript theme={null}
0 < {_} < 1
```
integer.
Values must be in the following range:
```javascript theme={null}
1 ≤ {_} < inf
```
The language of inputs texts.
If all texts are in the same language, it can be specified here instead of passing it as an input column. The language will be used to identify the correct spaCy model to parse and analyze the texts. For allowed values, see the comment regarding the `lang` column above.
N-gram configuration.
Configure maximum size, which words/tokens to exclude, and how to represent the n-grams in the result.
N-grams with up to this number of words will be extracted.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_max ≤ 4
```
Exclude these kinds of tokens from n-grams.
For longer n-grams, those containing either extraneous whitespace or any punctuation are automatically excluded. Additionally,
if `stops` is included in filters, n-grams containing stopwords as the first and/or last token are also excluded.
Each item in array.
Values must be one of the following:
`punct` `stops` `url` `digits` `non_alpha` `non_ascii`
Representation of the individual words/tokens to extract.
I.e. whether verbatim (text/ortho), lower(-case) or lemmatized. Also see spaCy's attribute reference
in [this table](https://spacy.io/api/token#attributes) for further information.
Values must be one of the following:
* `orth`
* `lemma`
* `lower`
* `text`
Whether unigrams should always be extracted lemmatized, irrespective of the `attrib` parameter.
Whether to separate the words in n-grams with an underscore character instead of a space.
N-gram frequency filter.
Filters n-grams based on the number of texts they occur in.
Minimum number of rows.
N-grams not occurring in at least these many rows (texts) will be excluded.
Values must be in the following range:
```javascript theme={null}
0 ≤ min_rows < inf
```
Maximum proportion of rows.
N-grams occurring in more than this proportion of rows (texts) will be excluded.
Values must be in the following range:
```javascript theme={null}
0 ≤ max_rows ≤ 1
```
Keep n most frequent keywords.
Whether to always include the n most frequent n-grams, independent of the other filter parameters.
Set to `null` to ignore.
Values must be in the following range:
```javascript theme={null}
0 ≤ keep_top_n < inf
```
Exclude n most frequent keywords.
Exclude the n most frequent n-grams, even if they passed the other filter conditions.
Values must be in the following range:
```javascript theme={null}
0 ≤ filter_top_n < inf
```
Filter per language.
Apply filter conditions separately to texts grouped by language, rather than across all texts.
# extract_range
Source: https://docs.graphext.com/api-docs/prepare/transform/extract_range
Create a copy of a column nullifying values outside a specified range.
For columns containing dates, numbers or lists of numbers, values from inside a list are removed, nullifying the list itself when becoming empty.
## Usage
The following example shows how the step can be used in a recipe.
Replace values outside of range \[rangeLeft, rangeRight) with nulls:
```stan theme={null}
extract_range(ds.price, {"rangeLeft": 200, "rangeRight": 800}) -> (ds.price_clipped)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
extract_range(col_in: date|number|list[number], {
"param": value,
...
}) -> (col_out: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column to extract the range from.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Inclusive left boundary of the selection's range.
* 200
* -0.7
* 2021-05-07T13:17:53Z
Exclusive right boundary of the selection's range.
* 800
* -0.2
* 2022-07-07T12:15:03Z
# extract_regex
Source: https://docs.graphext.com/api-docs/prepare/transform/extract_regex
Extract parts of texts detected using regular expressions.
A *regular expression* (or *regex*, *regex pattern*) is a sequence of characters that forms a search pattern.
This pattern is compared against texts, and any matches returned. The matches don't have to be returned as found,
but can be formatted using the `output` parameter. Check below references to familiarize yourself with the regex
language:
* [Google-RE2 regex wiki](https://github.com/google/re2/wiki/Syntax)
* [Wikipedia](https://en.wikipedia.org/wiki/Regular_expression)
Also see the `pattern` parameter below for more details.
## Usage
The following example shows how the step can be used in a recipe.
Extract all twitter mentions with handles between 1 and 15 characters long into lists of mentions
```stan theme={null}
extract_regex(ds.text, {
"pattern": "@\\w{1,15}",
"extract_all": true
}) -> (ds.mentions)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
extract_regex(text: text|category, {
"param": value,
...
}) -> (text_extracted: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A text column to extract parts from.
A column containing the extracted part (or parts) for each text. The column's data type will depend on the input
and specified parameters:
* Output column has type *text* when:
`"concat_matches": true` or `"extract_all": false` (matches are strings), and `"as_category": false`
* Output column has type *category* when:
`"concat_matches": true` or `"extract_all": false` (matches are strings), and `"as_category": true`
* Output column has type *list\[category]* when:
`"extract_all": true` and `"concat_matches": false`
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
A regular expression.
The pattern to be matched in input texts. May include (numbered) [regex capturing groups](https://www.regular-expressions.info/replacebackref.html),
which allows this method to use parts of a match to format the way matches are represented in the output via the
`output` parameter. The latter uses google-re2 string replacement with curly braces and numerical identifiers,
e.g. "" instead of the usual regex syntax using backslashes, like "\1". Numerical identifiers refer to capturing
groups in the regex pattern (named groups are not supported), where
* 0 is the whole match
* 1 is the 1st capturing group
* 2 is the 2nd capturing group
* etc...
The default is `"{0}"`, i.e. simply returning the full match.
For example, if a column of texts includes twitter mentions of the form "@abc", the regular expression
`"pattern": "(@)(\\w*)"`
will match these mentions and save the "@" character and the actual name in two separate capturing groups.
Using the output format
`"output": "Match: {0}, Tag: {1}, Name: {2}"`
will then return matches in the form "Match: @abc, Tag: @, Name: abc".
* @\w
Output format string.
Determines how matches will be represented in the output. Use numbers in curly braces to refer to captured groups.
Match criteria.
[Python-style regex flags](https://github.com/google/re2/wiki/Syntax#Flags) determining how to match.
Each item in array.
Values must be one of the following:
`ascii` `a` `ignorecase` `i` `locale` `l` `multiline` `m` `dotall` `s`
Whether to extract first match only or all matches (as lists).
Whether to concatenate all matches into a single text string.
The character (or string of characters) to use when concatenating multiple matches.
Whether to return a categorical rather than text column.
When the result would be text strings rather than lists (`"extract_all": false"` or `"concat_matches": false`),
whether to return a column of type *category* rather than *text*.
# extract_text_features
Source: https://docs.graphext.com/api-docs/prepare/transform/extract_text_features
Parse and process texts to extract multiple features at once.
Essentially combines all of the following steps into one:
* `embed_text`
* `extract_emoji`
* `extract_entities`
* `extract_hashtags`
* `extract_keywords`
* `extract_mentions`
* `infer_sentiment`
* `tokenize`
Note that the step does not currently allow for detailed configuration of each of the extracted features.
To do that, use any or all of the individual steps above.
## Usage
The following examples show how the step can be used in a recipe.
Extract all text features with automatic language detection
```stan theme={null}
extract_text_features(ds.text) => (ds.sentiment, ds.embedding, ds.hashtags, ds.mentions, ds.keywords, ds.tokens, ds.emoji, ds.people, ds.groups, ds.organizations, ds.gpes, ds.locations, ds.products, ds.events, ds.money)
```
Extract text features providing a language column
```stan theme={null}
extract_text_features(ds.text, ds.language) => (ds.sentiment, ds.embedding, ds.hashtags, ds.mentions, ds.keywords, ds.tokens, ds.emoji, ds.people, ds.groups, ds.organizations, ds.gpes, ds.locations, ds.products, ds.events, ds.money)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
extract_text_features(text: text, *lang: category, {
"param": value,
...
}) -> (
Sentiment: number,
Embedding: list[number],
Hashtags: list[category],
Mentions: list[category],
Keywords: list[category],
Tokens: list[category],
Emoji: list[category],
People: list[category],
Groups: list[category],
Organizatons: list[category],
GPEs: list[category],
Locations: list[category],
Products: list[category],
Events: list[category],
Money: list[category]
)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A text column to extract n-grams from.
An (optional) column identifying the languages of the corresponding texts. It is used to identify the correct model (spaCy)
to use for each text. If the dataset doesn't contain such a column yet, it can be created using the `infer_language` step.
Ideally, languages should be expressed as two-letter
[ISO 639-1 language codes](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes), such as "en", "es" or "de" for
English, Spanish or German respectively. We also detect fully spelled out names such as "english", "German", "allemande"
etc., but it is not guaranteed that we will recognize all possible spellings correctly always, so ISO codes should be
preferred.
Alternatively, if all texts are in the same language, it can be identified with the `lang` *parameter* instead.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
This step doesn't expect any specific parameters.
# extract_url_components
Source: https://docs.graphext.com/api-docs/prepare/transform/extract_url_components
Extract components from an URL.
Let's say we have `http://www.cwi.nl:80/%7Eguido/Python.html;a=2;b=3?c=4,2&d=e#anchor` as our URL.
Then these components will be the following:
* `scheme`: URL scheme specifier (http)
* `domain`: Network location part ([www.cwi.nl:80](http://www.cwi.nl:80))
* `path`: Hierarchical path (/%7Eguido/Python.html)
* `params`: Parameters for last path element (a=2;b=3)
* `query`: Query component (c=4,2\&d=e)
* `fragment`: Fragment identifier (anchor)
For more information about these components you can check urllib's description [here](https://docs.python.org/3/library/urllib.parse.html#url-parsing).
## Usage
The following example shows how the step can be used in a recipe.
Use `http` as default scheme.
```stan theme={null}
extract_url_components(ds.urls, {
"default_scheme": "http",
}) -> (
ds.scheme,
ds.domain,
ds.path,
ds.params,
ds.query,
ds.fragment
)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
extract_url_components(urls: url, {
"param": value,
...
}) -> (
scheme: category,
domain: category,
path: category,
params: category,
query: category,
fragment: category
)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
The list of URLs you wish to decomponse.
URL scheme specifier (http).
Network location part ([www.cwi.nl:80](http://www.cwi.nl:80)).
Hierarchical path (/%7Eguido/Python.html).
Parameters for last path element (a=2;b=3).
Query component (c=4,2\&d=e).
Fragment identifier, like after hashtag (anchor).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
URL Default Scheme.
If you wish to add a scheme (http, https...) prefix to those urls that don't have one, do it here.
If you wish none to be added, use null instead.
# Transform
Source: https://docs.graphext.com/api-docs/prepare/transform/index
| Step | Fast | Description |
| ------------------------------------------------------------------------------------------------------ | ---- | ----------------------------------------------------------------------------------------------------------------------- |
| [add\_noise](/api-docs/prepare/transform/add_noise) | | Add noise to a column with numbers or lists of numbers |
| [calculate](/api-docs/prepare/transform/calculate) | | Evaluates a formula containing basic arithmetic over a dataset's columns |
| [cast](/api-docs/prepare/transform/cast) | ⚡ | Interprets and changes a column's data to another (semantic) type |
| [concatenate](/api-docs/prepare/transform/concatenate) | ⚡ | Concatenate columns as text or lists with optional separator as well as pre- and postfix |
| [count\_unique](/api-docs/prepare/transform/count_unique) | ⚡ | Counts the number of unique elements in each list/array of the input column |
| [derive\_column](/api-docs/prepare/transform/derive_column) | ⚡ | Derive a new column with a custom JS script |
| [discretize\_on\_quantiles](/api-docs/prepare/transform/discretize_on_quantiles) | ⚡ | Discretize column into bins based on quantiles |
| [discretize\_on\_values](/api-docs/prepare/transform/discretize_on_values) | ⚡ | Discretize column by binning its values using explicitly specified cuts points |
| [divide](/api-docs/prepare/transform/divide) | ⚡ | Divide two or more numeric columns in given order |
| [equal](/api-docs/prepare/transform/equal) | ⚡ | Check the row-wise equality of all input columns |
| [explode](/api-docs/prepare/transform/explode) | ⚡ | Explode (extract) items from column(s) of lists into separate rows |
| [extract\_date\_component](/api-docs/prepare/transform/extract_date_component) | ⚡ | Extract a component such as day, week, weekday etc. from a date column |
| [extract\_emoji](/api-docs/prepare/transform/extract_emoji) | | Parse texts and extract their emoji |
| [extract\_entities](/api-docs/prepare/transform/extract_entities) | | Parse texts and extract the entities mentioned (persons, organizations etc.) |
| [extract\_hashtags](/api-docs/prepare/transform/extract_hashtags) | | Parse texts and extract any hashtags mentioned |
| [extract\_json\_values](/api-docs/prepare/transform/extract_json_values) | ⚡ | Extract values from JSON columns using JsonPath |
| [extract\_keywords](/api-docs/prepare/transform/extract_keywords) | | Parse and extract keywords from texts |
| [extract\_mentions](/api-docs/prepare/transform/extract_mentions) | | Parse texts and extract any mentions detected |
| [extract\_ngrams](/api-docs/prepare/transform/extract_ngrams) | | Parse texts and extract their n-grams |
| [extract\_range](/api-docs/prepare/transform/extract_range) | ⚡ | Create a copy of a column nullifying values outside a specified range |
| [extract\_regex](/api-docs/prepare/transform/extract_regex) | ⚡ | Extract parts of texts detected using regular expressions |
| [extract\_text\_features](/api-docs/prepare/transform/extract_text_features) | | Parse and process texts to extract multiple features at once |
| [extract\_url\_components](/api-docs/prepare/transform/extract_url_components) | | Extract components from an URL |
| [is\_missing](/api-docs/prepare/transform/is_missing) | ⚡ | Check for missing values in a given column |
| [label\_bios](/api-docs/prepare/transform/label_bios) | | Categorize people into fields of occupation using their bios (biographies) |
| [label\_categories](/api-docs/prepare/transform/label_categories) | ⚡ | Relabel categories based on the top terms in each category |
| [label\_encode](/api-docs/prepare/transform/label_encode) | | Encode categories with values between 0 and N-1, where N is the number of unique categories |
| [label\_holidays](/api-docs/prepare/transform/label_holidays) | | Indicate if there are any holidays for given date, location pairs |
| [label\_political\_subtopics](/api-docs/prepare/transform/label_political_subtopics) | | Categorize the political sub-topics of texts in Spanish |
| [label\_political\_topics](/api-docs/prepare/transform/label_political_topics) | | Categorize the political topics of texts in Spanish |
| [label\_texts\_containing](/api-docs/prepare/transform/label_texts_containing) | | Categorize texts containing specific keywords with custom labels |
| [label\_texts\_containing\_from\_query](/api-docs/prepare/transform/label_texts_containing_from_query) | | Label texts given an elastic-like query string |
| [length](/api-docs/prepare/transform/length) | ⚡ | Calculates the length of lists (number of elements) or texts/categories (number of characters) |
| [make\_constant](/api-docs/prepare/transform/make_constant) | ⚡ | Creates a new constant column (with a single unique value) of the same length as the input column |
| [math\_func](/api-docs/prepare/transform/math_func) | | Applies a mathematical function to the values of a (single) numeric column |
| [merge\_similar\_semantics](/api-docs/prepare/transform/merge_similar_semantics) | | Group categories with similar meanings |
| [merge\_similar\_spellings](/api-docs/prepare/transform/merge_similar_spellings) | | Group categories with similar spellings |
| [multiply](/api-docs/prepare/transform/multiply) | ⚡ | Multiply two or more numeric columns |
| [normalize](/api-docs/prepare/transform/normalize) | ⚡ | Normalizes a numerical column by subtracting the mean and dividing by its standard deviation |
| [observed\_duration](/api-docs/prepare/transform/observed_duration) | ⚡ | Calculate the duration between two dates and determine whether an event was observed before a specified observation da… |
| [order\_categories](/api-docs/prepare/transform/order_categories) | ⚡ | (Re-)order the categories of a categorical column |
| [pandas\_func](/api-docs/prepare/transform/pandas_func) | | Applies an arbitrary pandas supported function to the values of an input column |
| [pct\_change](/api-docs/prepare/transform/pct_change) | | Calculate percentage change between consecutive numbers in a numeric column |
| [percentile\_rank](/api-docs/prepare/transform/percentile_rank) | ⚡ | Convert the values in a numeric or date column into their percentile rank |
| [query](/api-docs/prepare/transform/query) | ⚡ | Generate a boolean column based on a query string, marking rows that match the condition |
| [replace\_missing](/api-docs/prepare/transform/replace_missing) | ⚡ | Replace missing values (NaNs) with either a specified constant value or the result of a given function |
| [replace\_regex](/api-docs/prepare/transform/replace_regex) | ⚡ | Replace parts of text detected with a regular expression |
| [replace\_values](/api-docs/prepare/transform/replace_values) | ⚡ | Replace specified values in a column with new ones |
| [scale](/api-docs/prepare/transform/scale) | ⚡ | Scales the values of a numerical column to lie between a specified minimum and maximum |
| [segment\_rows](/api-docs/prepare/transform/segment_rows) | ⚡ | Create a segmentation using graphext's advanced query syntax (similar to Elasticsearch) |
| [slice](/api-docs/prepare/transform/slice) | ⚡ | Extract a range/slice of elements from a column of texts or lists |
| [split\_string](/api-docs/prepare/transform/split_string) | ⚡ | Split a single column containing texts into two |
| [subtract](/api-docs/prepare/transform/subtract) | ⚡ | Subtract two or more numeric columns |
| [sum](/api-docs/prepare/transform/sum) | ⚡ | Calculate the row-wise sum of numeric columns |
| [time\_interval](/api-docs/prepare/transform/time_interval) | ⚡ | Calculates the duration of a time interval between two dates (datetimes/timestamps) |
| [tokenize](/api-docs/prepare/transform/tokenize) | | Parse texts and separate them into lists of tokens (words, lemmas, etc.) |
| [trim\_frequencies](/api-docs/prepare/transform/trim_frequencies) | | Remove values whose frequencies (counts) are above/below a given threshold |
| [unique](/api-docs/prepare/transform/unique) | ⚡ | Extracts the unique elements in each list/array |
| [unpack\_list](/api-docs/prepare/transform/unpack_list) | | Unpack (extract) items from a column of lists into separate columns |
# is_missing
Source: https://docs.graphext.com/api-docs/prepare/transform/is_missing
Check for missing values in a given column.
This step checks each row of the input column to determine if the value is missing (null or NaN).
The result is a new boolean column, where each row indicates whether the corresponding element
in the input column is missing.
The step can work with single-valued and multi-valued columns, and the output can be configured
to be either boolean (true/false), numeric (0/1) or categorical (custom labels).
* For single-valued columns: Each row in the output column will be `true` if the corresponding
value in the input column is missing, and `false` otherwise.
* For multivalued columns: Each row in the output column will be `true` if the corresponding
sub-list in the input column is empty, and `false` otherwise.
## Usage
The following examples show how the step can be used in a recipe.
Check for missing values in a numeric column.
```stan theme={null}
is_missing(ds.numeric_col) -> (ds.numeric_col_missing)
```
Check for missing values in a text column and set output type to numeric.
```stan theme={null}
is_missing(ds.string_col, {"out_type": "number"}) -> (ds.string_col_missing)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
is_missing(column: column, {
"param": value,
...
}) -> (result: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
The input column to check for missing values.
The output column indicating the presence of missing values in the input column.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Output type.
The data type of the output column.
* 'boolean': Output is true/false indicating missing or not.
* 'number': Output is 0/1 indicating missing or not.
* 'category': Output is specified by params\["labels"]\["true"] and params\["labels"]\["false"].
Values must be one of the following:
* `boolean`
* `number`
* `category`
Labels for the true and false categories.
An object mapping the "true" and "false" categories to custom labels.
Label for the "true" category.
Label for the "false" category.
# label_bios
Source: https://docs.graphext.com/api-docs/prepare/transform/label_bios
Categorize people into fields of occupation using their bios (biographies).
The categorization is performed using a predefined lookup-table matching certain keywords
with associated fields of occupation. E.g. bios will be categorized as "journalists" if their
texts contain any of the following words: "periodista", "journalist", "journalism", "periodismo",
"news", "noticia", "noticias".
Possible categories currently are:
* journalists
* business
* developers
* marketing
* travel
* photography
* university
* seo
* blogging
* sports
* politics
* social sciences
* medical
* entertainment
* art design
* economics
* videogames.
## Usage
The following example shows how the step can be used in a recipe.
This step has no configuration parameters, so simply use
```stan theme={null}
label_bios(ds.text) -> (ds.field_of_occupation)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
label_bios(bios: text) -> (labels: list[category])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column containing biographies (e.g. from social network profiles).
A column containing one or more fields of occupation for each bio.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
This step doesn't expect any configuration.
# label_categories
Source: https://docs.graphext.com/api-docs/prepare/transform/label_categories
Relabel categories based on the top terms in each category.
This function enables the relabeling of category labels based on the most significant terms, or `top_terms`, within
each category. It takes two columns as inputs: one with the `old_labels`, which can be single or multi-valued categories,
and one with the `top_terms` for each data point. The replacement of the labels is influenced by the specified rank method,
which can be `TFIDF`, `BACKGROUND`, `FOREGROUND`, `UPLIFT`, `ORDINAL`, or `ALPHANUM`, and the number of top terms considered
(specified by `top_n`).
## Usage
The following examples show how the step can be used in a recipe.
To replace labels in a column of categories using TFIDF:
```stan theme={null}
label_categories(ds.old_labels, ds.top_terms) -> (ds.new_labels)
```
To replace labels in a column of categories using BACKGROUND:
```stan theme={null}
label_categories(ds.old_labels, ds.top_terms, {
rank_method: 'BACKGROUND',
top_n: 3
}) -> (ds.new_labels)
```
To replace labels in a column of categories using BACKGROUND and ascending order:
```stan theme={null}
label_categories(ds.old_labels, ds.top_terms, {
rank_method: 'BACKGROUND',
top_n: 3,
ascending: true
}) -> (ds.new_labels)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
label_categories(old_labels: category|list, top_terms: category|text|list, {
"param": value,
...
}) -> (new_labels: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column containing the old labels, which could be single-value or multi-value categories.
A column containing lists of top terms for each data point.
The output column. Its data type will depend on the 'old\_labels' input column type.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
The method used to rank the top terms.
Values must be one of the following:
`TFIDF` `BACKGROUND` `FOREGROUND` `UPLIFT` `ORDINAL` `ALPHANUM`
The number of top terms considered for each label.
Values must be in the following range:
```javascript theme={null}
1 ≤ top_n < inf
```
Whether the terms should be sorted in ascending order.
# label_encode
Source: https://docs.graphext.com/api-docs/prepare/transform/label_encode
Encode categories with values between 0 and N-1, where N is the number of unique categories.
In other words, the first category will be assigned the number 0 and the last category a value of N-1.
## Usage
The following example shows how the step can be used in a recipe.
Encode department names as numeric codes
```stan theme={null}
label_encode(ds.department) => (ds.department_encoded)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
label_encode(labels: category) -> (label_codes: number)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A categorical column.
A numeric column where labels have been replaced with values between 0 and N-1.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
This step doesn't expect any configuration.
# label_holidays
Source: https://docs.graphext.com/api-docs/prepare/transform/label_holidays
Indicate if there are any holidays for given date, location pairs.
Assigns holiday labels to a given date, location pair. The date must be expressed through a date column,
but the region can be indicated by either giving a country and region column, or indicating the country, region through
parameters. If only one location column is given, it's assumed to be the country.
For supported country/region identifiers, please check [this table](https://github.com/dr-prodigy/python-holidays#available-countries).
Since holidays essentially rely on manual or at best heuristical tagging,
keep in mind that the further you go into the future or into the past the more unreliable the data becomes,
and the same can be said for small countries.
You read more about our holiday data provider [here](https://github.com/dr-prodigy/python-holidays).
## Usage
The following examples show how the step can be used in a recipe.
The following labels California holidays in a set of dates
```stan theme={null}
label_holidays(ds.date, {
"country": "US",
"region": "CA"
}) -> (ds.california_holidays, ds.is_holiday)
```
The following labels several country/region combinations
```stan theme={null}
label_holidays(ds.date, ds.country, ds.region) -> (ds.holidays, ds.is_holiday)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
label_holidays(date_col: date, *location: category, {
"param": value,
...
}) -> (holidays: list[category], is_holiday: boolean)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Dates column to label.
Optional columns containing the location names (Counry, Region) if different countries or regions are present.
A column containing the labels assigned to each text.
Indicates wether a given date had some holidays in it or not.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Country to retrieve the holiday from.
Values must be one of the following:
`ABW` `AE` `AGO` `AO` `AR` `ARE` `ARG` `AT` `AU` `AUS` `AUT` `AW` `Angola` `Argentina` `Aruba` `Australia` `Austria` `BD` `BDI` `BE` `BEL` `BG` `BGD` `BI` `BLG` `BLR` `BR` `BRA` `BW` `BWA` `BY` `Bangladesh` `Belarus` `Belgium` `Botswana` `Brazil` `Bulgaria` `Burundi` `CA` `CAN` `CH` `CHE` `CHL` `CL` `CO` `COL` `CUW` `CW` `CZ` `CZE` `Canada` `Chile` `Colombia` `Croatia` `Curacao` `Czech` `Czechia` `DE` `DEU` `DJ` `DJI` `DK` `DNK` `DO` `DOM` `Denmark` `Djibouti` `DominicanRepublic` `ECB` `EE` `EG` `EGY` `ES` `ESP` `EST` `Egypt` `England` `Estonia` `EuropeanCentralBank` `FI` `FIN` `FR` `FRA` `Finland` `France` `GB` `GBR` `GE` `GEO` `GR` `GRC` `Georgia` `Germany` `Greece` `HK` `HKG` `HN` `HND` `HR` `HRV` `HU` `HUN` `Honduras` `HongKong` `Hungary` `IE` `IL` `IN` `IND` `IRL` `IS` `ISL` `ISR` `IT` `ITA` `Iceland` `India` `Ireland` `IsleOfMan` `Israel` `Italy` `JAM` `JM` `JP` `JPN` `Jamaica` `Japan` `KE` `KEN` `KOR` `KR` `Kenya` `Korea` `LT` `LTU` `LU` `LUX` `LV` `LVA` `Latvia` `Lithuania` `Luxembourg` `MA` `MEX` `MOR` `MOZ` `MW` `MWI` `MX` `MY` `MYS` `MZ` `Malawi` `Malaysia` `Mexico` `Morocco` `Mozambique` `NG` `NGA` `NI` `NIC` `NL` `NLD` `NO` `NOR` `NZ` `NZL` `Netherlands` `NewZealand` `Nicaragua` `Nigeria` `NorthernIreland` `Norway` `PE` `PER` `PL` `POL` `PRT` `PRY` `PT` `PTE` `PY` `Paraguay` `Peru` `Poland` `Polish` `Portugal` `PortugalExt` `RO` `ROU` `RS` `RU` `RUS` `Romania` `Russia` `SA` `SAU` `SE` `SG` `SGP` `SI` `SK` `SRB` `SVK` `SVN` `SWE` `SaudiArabia` `Scotland` `Serbia` `Singapore` `Slovak` `Slovakia` `Slovenia` `SouthAfrica` `Spain` `Sweden` `Switzerland` `TAR` `TR` `TUR` `Turkey` `UA` `UK` `UKR` `US` `USA` `Ukraine` `UnitedArabEmirates` `UnitedKingdom` `UnitedStates` `VEN` `VN` `VNM` `Venezuela` `Vietnam` `Wales` `YV` `ZA` `ZAF`
Region identifier.
Check table referenced above for valid values.
# label_political_subtopics
Source: https://docs.graphext.com/api-docs/prepare/transform/label_political_subtopics
Categorize the political sub-topics of texts in Spanish.
## Usage
The following example shows how the step can be used in a recipe.
Label Spanish texts with political subtopic categories
```stan theme={null}
label_political_subtopics(ds.text) => (ds.political_subtopics)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
label_political_subtopics(text_col: text) -> (labels: list[category])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A text column to label.
A column containing the labels assigned to each text.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
This step doesn't expect any configuration.
# label_political_topics
Source: https://docs.graphext.com/api-docs/prepare/transform/label_political_topics
Categorize the political topics of texts in Spanish.
## Usage
The following example shows how the step can be used in a recipe.
Label Spanish texts with political topic categories
```stan theme={null}
label_political_topics(ds.text) => (ds.political_topics)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
label_political_topics(text_col: text) -> (labels: list[category])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A text column to label.
A column containing the labels assigned to each text.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
This step doesn't expect any configuration.
# label_texts_containing
Source: https://docs.graphext.com/api-docs/prepare/transform/label_texts_containing
Categorize texts containing specific keywords with custom labels.
Assigns each text to one or more categories. Each category is defined by a list of keywords a text must
include or exclude to be labelled accordingly. In addition, each category may specify whether a keyword
must be matched explicitly, ignoring its case (lower, upper) etc. See parameters below for further details.
## Usage
The following example shows how the step can be used in a recipe.
The following defines the keywords to be included or exluded for each of three categories, labelled "journalist", "business" and "CEO". Note how in the case of "CEO" we're looking for occurrences of the spelling with capitals only.
```stan theme={null}
label_texts_containing(ds.text, {
"journalists": {
"include": ["journalist", "journalism", "news"],
"exclude": ["blogger"],
"case_sensitive": false
},
"business": {
"include":["startup", "entrepreneur", "founder"]
},
"CEOs": {
"include": ["CEO"],
"case_sensitive": true
}
}) -> (ds.field_of_occupation)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
label_texts_containing(text_col: text|category, {
"param": value,
...
}) -> (labels: list[category])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A text column to label.
A column containing the labels assigned to each text.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
One or more named text categories.
Each parameter should be a key indicating the name/label to show for a specific text category,
and should have an object as value specifying the terms a text must or must not contain for that
particular label to apply. Also see examples above.
List of strings a text must include to apply a label.
Each item in array.
List of strings a text must not include to apply a label.
Each item in array.
Whether to make search accent-sensitive.
Whether to make search case-sensitive.
Whether to match whole words only.
If enabled, only matches a word if it is surrounded by non-alphanumeric characters.
# label_texts_containing_from_query
Source: https://docs.graphext.com/api-docs/prepare/transform/label_texts_containing_from_query
Label texts given an elastic-like query string.
Given a query of the form "word1; word2 OR word3", texts containing "word1" will be labeled as
"word1", and texts containing "word2" or "word3" will be labeled as "word2 OR word3". In other words, each semicolon-separated
string acts as both query and corresponding label. Texts matching multiple queries will be assigned multiple labels.
## Usage
The following example shows how the step can be used in a recipe.
```stan theme={null}
label_texts_containing_from_query(ds.text, {"query": "startup OR entrepreneur; marketing OR -digital; devops"}) -> (ds.field_of_occupation)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
label_texts_containing_from_query(text_col: text|category, {
"param": value,
...
}) -> (labels: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A text column to label.
A column containing the labels assigned to each text.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Query to label.
Query is a string of labels/categories and associated keywords (see examples below).
Use ";" to separate categories, "OR" to join words for a category, and "-" to exclude words from a category.
The category label(s) will be formed using the query, e.g. a text containing "AA" and "BB" will be tagged as \[AA,BB].
* Cristiano OR -Five; for
Whether to make search accent sensitive.
Whether to make search case sensitive.
Whether to match whole words only.
If enabled, only matches a word if it is surrounded by non-alphanumeric characters.
Whether to return only the first match.
If True, only the first match will be assigned to each text. The result will be a simple categorical column.
If False, all identified matches will be assigned to each text. The result will be a multivalued column containing lists of categories.
# length
Source: https://docs.graphext.com/api-docs/prepare/transform/length
Calculates the length of lists (number of elements) or texts/categories (number of characters).
## Usage
The following example shows how the step can be used in a recipe.
This step has no parameters, so it's simply
```stan theme={null}
length(ds.input) -> (ds.input_lengths)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
length(input: list|text|category) -> (output: number)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column of input lists, texts or categories.
A column containing lengths of the input column's values.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
This step doesn't expect any configuration.
# make_constant
Source: https://docs.graphext.com/api-docs/prepare/transform/make_constant
Creates a new constant column (with a single unique value) of the same length as the input column.
## Usage
The following example shows how the step can be used in a recipe.
To force the language identifier of a dataset to be all Spanish:
```stan theme={null}
make_constant(ds.lang, {"value": "es"}) -> (ds.lang_es)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
make_constant(input: column, {
"param": value,
...
}) -> (output: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An arbitrary input column, used only to match its dimension (number of rows).
Column filled with copies of the value specified in parameters. The data type of the output column will be the one specified by out\_type.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
The value to fill the new column with.
The data type of the output column depends on this value and the `out_type` parameter.
A single string.
A single number.
A null value.
Array of strings.
Each item in array.
Array of numbers.
Each item in array.
Select types using their name.
Values must be one of the following:
`category` `date` `number` `boolean` `url` `sex` `text` `currency` `list[number]` `list[category]` `list[url]` `list[boolean]` `list[date]` `list[currency]`
# math_func
Source: https://docs.graphext.com/api-docs/prepare/transform/math_func
Applies a mathematical function to the values of a (single) numeric column.
See [numpy's documentation](https://numpy.org/doc/stable/reference/routines.math.html) for further details
about the supported functions.
## Usage
The following examples show how the step can be used in a recipe.
To calculate the logarithm of a column's values:
```stan theme={null}
math_func(ds.input, {"func": "log"}) -> (ds.output)
```
Or to calculate the absolute (distance from 0, |x|) of a column's values:
```stan theme={null}
math_func(ds.input, {"func": "absolute"}) -> (ds.output)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
math_func(input: number, {
"param": value,
...
}) -> (output: number)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Numeric input column to aplply function to.
The result of calling the specified function on the input data.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
The name of the mathematical function to apply.
Values must be one of the following:
`absolute` `angle` `arccos` `arccosh` `arcsin` `arcsinh` `arctan` `arctanh` `around` `cbrt` `ceil` `conj` `conjugate` `cos` `cosh` `cumprod` `cumsum` `deg2rad` `degrees` `exp` `exp2` `expm1` `fabs` `fix` `floor` `gradient` `i0` `imag` `log` `log10` `log1p` `log2` `nan_to_num` `nancumprod` `nancumsum` `nanprod` `nansum` `negative` `positive` `prod` `rad2deg` `radians` `real` `real_if_close` `reciprocal` `rint` `round` `round_` `sign` `sin` `sinc` `sinh` `sqrt` `square` `sum` `tan` `tanh` `trunc`
# merge_similar_semantics
Source: https://docs.graphext.com/api-docs/prepare/transform/merge_similar_semantics
Group categories with similar meanings.
This step calculates embeddings for each category using GloVe vectors provided by spaCy's models.
As similar words will have similar embeddings, we use them to cluster the categories, obtaining new
categories that groups the original ones.
## Usage
The following example shows how the step can be used in a recipe.
The following configuration applies the algorithm with the default values:
```stan theme={null}
merge_similar_semantics(ds.categories, ds.lang) -> (ds.new_categories)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
merge_similar_semantics(col: category|text|list[category], language: category, {
"param": value,
...
}) -> (categories: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Column with categories to merge.
Column containing merged categories.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Determines which categories will be merged.
After hierarchically clustering all categories, clusters of categories closer than this distance
will be merged into one.
Also see details in [scikit-learn's Agglomerative Clustering](https://scikit-learn.org/stable/modules/clustering.html#hierarchical-clustering).
Values must be in the following range:
```javascript theme={null}
0 ≤ distance_threshold ≤ 1
```
Which linkage criterion to use in the clustering.
While the distance metric applied is always the cosine between category embeddings, this parameter
determines how to calculate the distance between clusters of embeddings, e.g. selecting the maximum
distance between categories in two clusters ("complete"), the minimum ("single") etc.
Also see details in [scikit-learn's Agglomerative Clustering](https://scikit-learn.org/stable/modules/clustering.html#hierarchical-clustering).
Values must be one of the following:
* `single`
* `ward`
* `complete`
* `average`
# merge_similar_spellings
Source: https://docs.graphext.com/api-docs/prepare/transform/merge_similar_spellings
Group categories with similar spellings.
Real world texts are full of abbreviations, typos and slang, and so a single concept
can often be written in many different ways. For example, asking people to use free-form text
to indicate their job role in a questionaire may result in tens of different ways to describe
the same role (e.g. "Analista Programador", "analista / programador", "anilist y programmador" etc.).
This step attempts to clean up data of this type by merging categories with sufficiently similar spelling.
Behind the scenes the step uses [Chars2vec](https://github.com/IntuitionEngineeringTeam/chars2vec#chars2vec),
a library that employs recurrent neural networks to calculate character-based word embeddings. That is,
it transforms words into numeric vectors (embeddings) whose distance from another indicates the
similarity of the words' spelling.
We use *chars2vec* embeddings here to first identify sufficiently similar clusters of categories
(using [Agglomerative Clustering](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html)).
The output of the step then is simply a new column where each original category has been replaced by
the most common spelling in the same cluster.
## Usage
The following example shows how the step can be used in a recipe.
The following configuration applies the algorithm with the default values:
```stan theme={null}
merge_similar_spellings(ds.categories) -> (ds.merged_categories)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
merge_similar_spellings(col: category|text|list[category]|list[text], {
"param": value,
...
}) -> (categories: list[category])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Column with categories to merge.
Column containing merged categories.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Only words/categories with more characters than this will potentially be merged.
You may not want to merge short words, even if they're very similar (e.g. cat and cut).
Set this to 1 to potentially merge all categories.
Values must be in the following range:
```javascript theme={null}
1 ≤ min_length < inf
```
Only words/categories with fewer characters than this will potentially be merged.
Set this to null to include all categories in the algorithm.
Values must be in the following range:
```javascript theme={null}
1 ≤ max_length < inf
```
Whether or not to remove/convert all non-alphanumeric characters from categories before attempting to merge.
Any category label with a greater proportion of numeric digits than this will be excluded from merging.
Set to 1 to include all categories in the algorithm.
Values must be in the following range:
```javascript theme={null}
0 ≤ numeric_threshold ≤ 1
```
Split input texts into words at this character.
A string or regular expression identifying parts of text to be ignored when deciding which categories to merge.
Whether to penalize similarities depending on the length (#characters) of category labels.
The longer a category label, the less influence individual characters (and therefore small changes in spelling)
will have when comparing categories. This may lead to longer category labels being merged when they shouldn't.
Setting `"penalty": true` will make this less likely.
Maximum distance between groups of categories (embeddings) to be merged into the same cluster.
Also see parameters in [scikit-learn's Agglomerative Clustering](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html#sklearn-cluster-agglomerativeclustering).
Values must be in the following range:
```javascript theme={null}
0 ≤ distance_threshold < inf
```
Which linkage criterion to use in the clustering.
The distance measured between clusters of category embeddings to decide whether or not to merge them.
Also see parameters in [scikit-learn's Agglomerative Clustering](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html#sklearn-cluster-agglomerativeclustering).
Values must be one of the following:
* `single`
* `ward`
* `complete`
* `average`
Metric used to compute the distance between category embeddings in the clustering.
Also see parameters in [scikit-learn's Agglomerative Clustering](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html#sklearn-cluster-agglomerativeclustering).
Values must be one of the following:
* `euclidean`
* `l1`
* `l2`
* `manhattan`
* `cosine`
# multiply
Source: https://docs.graphext.com/api-docs/prepare/transform/multiply
Multiply two or more numeric columns.
An additional constant may be used to multiply the final result by.
## Usage
The following examples show how the step can be used in a recipe.
To multiply two input columns `factor1` and `factor2`:
```stan theme={null}
multiply(ds.factor1, ds.factor2) -> (ds.product)
```
To multiply a single column by a factor of 3.0:
```stan theme={null}
multiply(ds.num_column, {"constant": 3.0}) -> (ds.result)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
multiply(*columns: number, {
"param": value,
...
}) -> (result: number)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
One or more columns to multiply.
The result of the multiplication.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
A constant to multiply by.
Whether to ignore NaNs or propagate them.
# normalize
Source: https://docs.graphext.com/api-docs/prepare/transform/normalize
Normalizes a numerical column by subtracting the mean and dividing by its standard deviation.
The resulting column's values will have a mean of 0.0 and a standard deviation of 1.0.
Both types of scaling can be toggled separately via the `with_mean` and `with_std` parameters.
## Usage
The following examples show how the step can be used in a recipe.
To normalize using both mean and standard deviation:
```stan theme={null}
normalize(ds.input) -> (ds.normalized)
```
Using a custom configuration to only subtract the mean:
```stan theme={null}
normalize(ds.input, {
"with_mean": true,
"with_std": false,
}) -> (ds.normalized)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
normalize(input: number, {
"param": value,
...
}) -> (output: number)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A numeric column to normalize.
A numeric column containing the normalized value.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Whether to subtract the mean.
Whether to divide by the standard deviation.
# observed_duration
Source: https://docs.graphext.com/api-docs/prepare/transform/observed_duration
Calculate the duration between two dates and determine whether an event was observed before a specified observation date.
This step calculates the duration between a start date and an end date and determines whether an event was observed.
The output consists of two columns:
* `duration`: The time interval between the start and end dates in the specified unit (default: days).
* `observed`: A boolean column indicating whether the event was observed (i.e., if the end date occurs before the observation date).
This is particularly useful for preparing input data for survival analysis, such as Kaplan-Meier curves, where the event observation (censoring) status and duration are key inputs.
* If either `start_date` or `end_date` is missing (null), `observed` will be false, and `duration` will be null.
* Otherwise, the `duration` is calculated as the interval between `start_date` and `end_date`.
* If `end_date` is not null, `observed` will be true if `end_date <= observation_end`; otherwise, it will be false.
## Usage
The following examples show how the step can be used in a recipe.
Calculate the duration and observation status between a start date and end date.
```stan theme={null}
observed_duration(ds.start_date, ds.end_date, {"observation_end": "2020-01-01"}) -> (ds.duration, ds.observed)
```
Calculate the duration in weeks.
```stan theme={null}
observed_duration(ds.start_date, ds.end_date, {"observation_end": "2020-01-01", "unit": "weeks"}) -> (ds.duration, ds.observed)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
observed_duration(start_date: date, end_date: date, {
"param": value,
...
}) -> (duration: number, observed: boolean)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
The column containing the start date for each entry.
The column containing the end date for each entry (can be null for ongoing cases).
The calculated duration between the start and end dates in the specified unit.
A boolean column indicating if the event was observed before the `observation_end`.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Observation end date.
The cutoff date to determine if the event was observed (e.g., churn or any other event).
Unit for duration.
The unit of measurement for the duration. Allowed values are: - "Y", "year" - "Q", "quarter" - "M", "month" - "W", "week" - "D", "day" - "h", "hour" - "m", "minute" - "s", "second" - "ms", "millisecond"
The unit name can be spelled in singular or plural and is case-insensitive.
Values must be one of the following:
`Y` `year` `Year` `years` `Years` `Q` `quarter` `Quarter` `quarters` `Quarters` `M` `month` `Month` `months` `Months` `W` `week` `Week` `weeks` `Weeks` `D` `day` `Day` `days` `Days` `h` `hour` `Hour` `hours` `Hours` `m` `minute` `Minute` `minutes` `Minutes` `s` `second` `Second` `seconds` `Seconds` `ms` `millisecond` `Millisecond` `milliseconds` `Milliseconds`
# order_categories
Source: https://docs.graphext.com/api-docs/prepare/transform/order_categories
(Re-)order the categories of a categorical column.
The output, if transformation is successful, will always be an ordinal (ordered `category`) column,
even if the input was an unordered `category` or not a `Category` at all. I.e. it is supposed
that re-ordering the categories means the order is important.
If the input column is already a `category`:
* If unordered (non-ordinal): categories will be ordered in the given order (converted to ordinal)
* If ordered (ordinal): categories will be re-ordered only
In both cases, the specified categories have to match the ones already existing. I.e. only re-ordering
is allowed, but not deletion or addition of new categories.
If the column is *not* already a `category`:
* the column will be converted if the param `force_categorical` is `true`, and ordered as desired. Otherwise the new column will be identical to the input (no ordering performed).
## Usage
The following examples show how the step can be used in a recipe.
To arrange the categories "small", "medium", "large" ("S", "M", "L") in reverser order:
```stan theme={null}
order_categories(ds.cat_col, {"categories": ["L", "M", "S"]}) -> (ds.ordinal)
```
Converting a text column containing the strings "low", "medium" and "high" to an ordinal column:
```stan theme={null}
order_categories(ds.text, {
"categories": ["low", "medium", "high"],
"force_categorical": true
}) -> (ds.ordinal)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
order_categories(input: column, {
"param": value,
...
}) -> (output: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column to re-order or convert to ordinal (ordered `category`).
An ordinal column.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
List with desired order of categories.
If `null`, the unique and lexicographically sorted existing values in the input column will be used.
Each item in array.
Whether to convert non-categorical input columns to `category` before ordering (otherwise will be unchanged).
# pandas_func
Source: https://docs.graphext.com/api-docs/prepare/transform/pandas_func
Applies an arbitrary pandas supported function to the values of an input column.
Note, this is a somewhat advanced step. In particular, due to its generality, its parameters will not be
validated before execution, and so it is possible to call this step with parameters that will lead to
failure.
The function to be applied must be accesible as a method of a pandas `Series`.
For further detail see the corresponding [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.html).
However, only functions compatible with the column's type should be used (not e.g. the function `sum` when
the input column contains texts). To ensure the correct type given a desired function, you may cast the input
column to a different type before applying the function (see the `in_type` parameter below).
Some additional functions specific to datetime, text and categorical columns are available under pandas'
`dt`, `str`, and `cat` [accessors](https://pandas.pydata.org/pandas-docs/stable/reference/series.html#accessors).
See the `acc` parameter below.
Also, any function available in numpy's or pandas' global namespace (i.e. as `np.func` or `pd.func`), and which
transform a singe element (rather than a whole column), may be applied to the elements of the input using
`apply` as the `func` parameter, and the name of a specific function as the `elem_func` parameter.
Finally, the result of applying the desired function can be forced to a specific output type using the
`out_type` parameter.
See below examples for usage in the different scenarios.
## Usage
The following examples show how the step can be used in a recipe.
To create a column indicating whether a value in the input is missing or not
```stan theme={null}
pandas_func(ds.input, {"func": "isna"}) -> (ds.output)
```
Using a custom configuration to apply the `np.log` function to all (non-NaN) elements of a numeric(!) input column:
```stan theme={null}
pandas_func(ds.input, {
"func": "apply",
"elem_func": "np.log",
}) -> (ds.log_var)
```
Cast a numeric input column to text, then use Pandas' `str` accessor to get the length of the number strings, i.e. the number of characters.
```stan theme={null}
pandas_func(ds.input, {
"in_type": "Text",
"acc": "str",
"func": "len",
"out_type": "number"
}) -> (ds.number_digits)
```
To calculate the length of lists (note that in this case it would be simpler and better to use the dedicated step `length`):
```stan theme={null}
pandas_func(ds.input_lists, {
"func": "apply",
"elem_func": "np.size"
}) -> (ds.list_length)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
pandas_func(input: column, {
"param": value,
...
}) -> (output: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An arbitrary input column. But, note that the pandas function to be called has to be compatible with the data type of the input column!
The result of calling the specified Pandas function on the input series.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
The name of a pandas function to be applied.
Must be accesible as a method of a pandas [Series object](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.html).
The semantic type to cast the input column to *before* calling the specified function `func`.
Values must be one of the following:
`Category` `Date` `Number` `Boolean` `Url` `Sex` `Text` `List[Number]` `List[Category]` `List[Url]` `List[Boolean]` `List[Date]` `number` `boolean` `url` `sex` `text` `list[number]` `list[category]` `list[url]` `list[boolean]` `list[date]`
The semantic type to cast the result to *after* calling the specified function `func`.
Values must be one of the following:
`category` `date` `number` `boolean` `url` `sex` `text` `list[number]` `list[category]` `list[url]` `list[boolean]` `list[date]`
A pandas accessor used on the input column before calling the specified function `func`.
For further information see [accessors](https://pandas.pydata.org/pandas-docs/stable/reference/series.html#accessors).
Values must be one of the following:
* `str`
* `dt`
* `cat`
When `func` is `apply`, the name of a function to be applied to the *elements* of the input column.
# pct_change
Source: https://docs.graphext.com/api-docs/prepare/transform/pct_change
Calculate percentage change between consecutive numbers in a numeric column.
For a pair of consecutive numbers x1, x2 in the input column, calculates the difference between x2 and x1
expressed as a percentage of x1, i.e. `100 * (x2 - x1) / x1`.
## Usage
The following examples show how the step can be used in a recipe.
Calculate daily percentage change of stock prices sorted by date
```stan theme={null}
pct_change(ds.price, ds.date) => (ds.price_pct_change)
```
Calculate proportional change in revenue without sorting
```stan theme={null}
pct_change(ds.revenue, {"as_proportion": true}) => (ds.revenue_change)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
pct_change(data: number, *sort: column, {
"param": value,
...
}) -> (output: number)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A numeric column to calculate percentage change for.
An optional column used to sort the rows before calculating the percent change.
A numeric column with the calculated percentages.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Method to fill missing values (NaNs).
When `forward`, uses last valid observation to fill a gap. When `backwards`, uses next valid observation instead.
Values must be one of the following:
* `forward`
* `backward`
Maximum number of consecutive NaNs to fill before stopping.
Values must be in the following range:
```javascript theme={null}
1 ≤ fill_nan_max < inf
```
Whether to express result as proportion instead of percentage.
Whether to sort ascending instead of descending using the `sort` column before calculating the change.
# percentile_rank
Source: https://docs.graphext.com/api-docs/prepare/transform/percentile_rank
Convert the values in a numeric or date column into their percentile rank.
This function calculates the percentile rank for each non-null value in a single-valued numeric or date column.
Percentile ranks are assigned based on the relative position of each value in the sorted column, with results ranging
\[0, 1). Null values are preserved and do not affect the ranking.
The function does not support multi-valued columns.
## Usage
The following example shows how the step can be used in a recipe.
The following example calculates the percentile rank for a column of numerical values, producing a new numerical column where each value is ranked between 0 and 1.
```stan theme={null}
percentile_rank(ds.age, {}) -> (ds.age_percentile)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
percentile_rank(input: number, {
"param": value,
...
}) -> (output: number)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A single-valued numeric or date column for percentile ranking.
A new numerical column containing the percentile ranks for each value.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
This step expects not to receive any parameters.
# query
Source: https://docs.graphext.com/api-docs/prepare/transform/query
Generate a boolean column based on a query string, marking rows that match the condition.
## Usage
The following example shows how the step can be used in a recipe.
This example creates a new boolean column that flags rows where the 'cats' column contains the value 'red':
```stan theme={null}
query(ds, {"query": "cats: red"}) -> (ds.query_result)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
query(ds_in: dataset, {
"param": value,
...
}) -> (result: boolean)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
The input dataset to evaluate against the query.
A boolean column indicating whether each row in the dataset matches the query.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
The *graphext advanced query* used to identify rows to flag.
# replace_missing
Source: https://docs.graphext.com/api-docs/prepare/transform/replace_missing
Replace missing values (NaNs) with either a specified constant value or the result of a given function.
## Usage
The following examples show how the step can be used in a recipe.
The following configuration fills all missing values with the string "unknown":
```stan theme={null}
replace_missing(ds.occupation, {"value": "unknown"}) -> (ds.occupation_filled)
```
The following configuration fills all missing values with the maximum of the column:
```stan theme={null}
replace_missing(ds.numbers, {"function": "max"}) -> (ds.numbers_filled)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
replace_missing(input: column, {
"param": value,
...
}) -> (output: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An arbitrary column, potentially containing missing values (NaN).
A copy of the input column where missing values have been replaced by a constant.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
The constant to use to fill in missing values (normally of same type as original column).
Can be a scalar value (with number or string type) or an array of values (number or string). If an array
is passed it should have at least one item.
Each item in array.
Fill missing values with the result of a given function.
The following functions can be used:
* max: substitutes the NaN values with the maximum value of a numerical column.
* min: substitutes the NaN values with the minimum value of a numerical column.
* mean: substitutes the NaN values with the mean of a numerical column.
* median: substitutes the NaN values with the median of a numerical column.
* least\_freq: substitutes the NaN values with the least frequent value of a column.
* most\_freq: substitutes the NaN values with the most frequent value of a column.
* alphabetical\_first: substitutes the NaN values with the alphabetically first value of a categorical column.
* alphabetical\_first: substitutes the NaN values with the alphabetically last value of a categorical column.
* bfill: for each NaN value, uses the next valid observation to fill it.
* ffill: for each NaN value, propagates the last valid observation forward to fill it.
Values must be one of the following:
`max` `min` `mean` `median` `least_freq` `most_freq` `alphabetical_first` `alphabetical_last` `bfill` `ffill`
# replace_regex
Source: https://docs.graphext.com/api-docs/prepare/transform/replace_regex
Replace parts of text detected with a regular expression.
A regular expression (or regex, regex pattern) is a sequence of characters that forms a search pattern. This
pattern is compared against texts, and any matches are substituted by a desired replacement. The replacement
can be a simple (constant) text string, or a formatting pattern referencing all or parts of the matched character
sequence.
Simple replacement of fixed text strings with another fixed text string can be performed easily. E.g., to
replace all occurrences of "hi" with "hello", you'd simply use `{"pattern": "hi", "replacement": "hello"}`.
However, using capturing groups in `pattern` and `replacement` parameters allows for much greater flexibility.
For example, if a column of texts includes twitter mentions of the form "@abc", the regular expression
`"pattern": "@(\\w*)"` will match these mentions and save the actual name without the "@" character in a capturing
group. Using the replacement string `"replacement": "{1}"` will then replace all matched mentions with only the name
part of the twitter handle, effectively removing the "@" tags from all mentions (without removing other occurrences of
the "@" character).
To further familiarize yourself with the regex language also see these references:
* [Google-RE2 regex wiki](https://github.com/google/re2/wiki/Syntax)
* [Wikipedia](https://en.wikipedia.org/wiki/Regular_expression).
## Usage
The following example shows how the step can be used in a recipe.
To change the way dates are formatted in a column of texts from "2019-04-15" to "15.04.2019":
The specified pattern will match 3 numbers separated by the minus sign, and will replace such occurences by the same
three numbers in reverse order and separated with a period.
```stan theme={null}
replace_regex(ds.text, {
"pattern": "(\d+)-(\d+)-(\d+)",
"replacement": "{3}.{2}.{1}"
}) -> (ds.replaced)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
replace_regex(text: text|category|list, {
"param": value,
...
}) -> (replaced: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column containing text-like values.
The output column's data type will depend on the input and specified parameters:
* `text`: if input is text and parameter `"as_category": false`
* `category`: if input is *not* a column of lists and `"as_category": true`
* `list`: if input is a column of lists.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Regular expression to be matched in your input texts.
The regex `pattern` may include (numbered) [regex capturing groups](https://www.regular-expressions.info/replacebackref.html),
which allows this method to use parts of a match to format the way matches are then replaced in the output via the `replacement`
parameter (see below).
A format string determining how matches will be replaced in the text.
The replacement string allows for use of Google-RE2 style string replacement with curly braces and numerical identifiers, e.g. "" instead
of the usual regex syntax using backslashes ("\1"). Numerical identifiers refer to capturing groups in the regex pattern (named groups are
not supported), where
* 0 is the whole match
* 1 is the 1st capturing group
* 2 is the 2nd capturing group
* etc...
The default is the empty string "", i.e. matched parts will be removed from the text.
Regex configuration flags.
Uses [Python-style regex flags](https://github.com/google/re2/wiki/Syntax#Flags) determining how to perform matches, e.g. \["a", "ignorecase"]
Note that flags not present in Google-RE2 implementation will be ignored.
Each item in array.
Values must be one of the following:
`a` `ascii` `i` `ignorecase` `l` `locale` `m` `multiline` `s` `dotall`
Whether to cast result to category data type when otherwise it would be texts.
# replace_values
Source: https://docs.graphext.com/api-docs/prepare/transform/replace_values
Replace specified values in a column with new ones.
This function enables the replacement of specified values in a column with new ones. It takes a mapping object, where
each key-value pair represents an "old value" -> "new value" transformation. The function scans the column for values
that match any of the keys in the mapping object and replaces them with their corresponding new values.
This function is case-sensitive and performs exact matches.
For numeric replacements, it's important to note that keys in the mapping object need to be strings, even for numeric
columns. The function will internally convert these keys into numbers for comparison.
## Usage
The following examples show how the step can be used in a recipe.
To change specific names in a column of texts from "pedro" to "pablo" and "maria" to "mariana":
```stan theme={null}
replace_values(ds.names, {
"pedro": "pablo",
"maria": "mariana"
}) -> (ds.replaced)
```
To change specific numbers in a column of numerical values from "20" to 99 and "30" to 100:
```stan theme={null}
replace_values(ds.age, {
"20": 99,
"30": 100
}) -> (ds.replaced)
```
To replace null values in a column with a string:
```stan theme={null}
replace_values(ds.column, {
"": "something not null"
}) -> (ds.replaced)
```
To replace a specific numerical value in a column with null using the keyword null:
```stan theme={null}
replace_values(ds.age, {
"30": null
}) -> (ds.replaced)
```
To replace a specific numerical value in a column with null using the string "null":
```stan theme={null}
replace_values(ds.age, {
"30": "null"
}) -> (ds.replaced)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
replace_values(original: category|text|number|date|list, {
"param": value,
...
}) -> (replaced: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column containing text-like values.
The output column's data type will depend on the input and specified parameters:
* `text`: if input is text and parameter `"as_category": false`
* `category`: if input is *not* a column of lists and `"as_category": true`
* `list`: if input is a column of lists, `list` of the same kind as the input.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
One or more additional parameters.
# scale
Source: https://docs.graphext.com/api-docs/prepare/transform/scale
Scales the values of a numerical column to lie between a specified minimum and maximum.
Without further specification, the minimum and maximum by default are 0.0 and 1.0 respectively.
## Usage
The following examples show how the step can be used in a recipe.
The following example scales input values to be in the range \[0, 1].
```stan theme={null}
scale(ds.input) -> (ds.scaled)
```
Using a custom configuration to scale values to the interval \[-2, 2], rather than the default of \[0, 1]:
```stan theme={null}
scale(ds.input, {
"min": -2,
"max": 2,
}) -> (ds.scaled)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
scale(input: number, {
"param": value,
...
}) -> (output: number)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A numeric input column.
A numeric column containing the scaled values.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
The minimum value after scaling.
The maximum value after scaling.
# segment_rows
Source: https://docs.graphext.com/api-docs/prepare/transform/segment_rows
Create a segmentation using graphext's advanced query syntax (similar to Elasticsearch).
## Usage
The following examples show how the step can be used in a recipe.
This simple query creates a new segmentation differentiating between adults and minors:
```stan theme={null}
segment_rows(ds, {"adult": "age: >=18", "minor": "age: <18"}) -> (ds.segmentation)
```
Flag clients that are exactly 19 years old and 21 years old
```stan theme={null}
segment_rows(ds, {"nineteen": "age:19", "twentyone": "age:21"}) -> (ds.segmentation)
```
Bin competitors who are over 27 years of age but below the mean age
```stan theme={null}
segment_rows(ds, {"over_twentyseven_below_mean": "age:>27 AND < MEAN"}) -> (ds.segmentation)
```
Select all who belong to the cool class
```stan theme={null}
segment_rows(ds, {"cool_clients": "class: cool"}) -> (ds.segmentation)
```
Segment all belonging to the most frequent 4 classes and least frequent 3
```stan theme={null}
segment_rows(ds, {"most_freq": "class: TOP(4)", "least_freq": "class: BOTTOM(3)"}) -> (ds.segmentation)
```
For those aged 18, separate on who earn more than 5 dollars monthly on average and 5 or less
```stan theme={null}
segment_rows(ds, {"high_inc": "(age: 18) AND ('avg monthly income':>5)", "low_inc": "(age: 18) AND ('avg monthly income':<=5)"}) -> (ds.segmentation)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
segment_rows(ds_in: dataset, {
"param": value,
...
}) -> (segmentation: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An input dataset to use for creating a segmentation.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
The *graphext advanced query* used to identify the rows to include in each segment.
# slice
Source: https://docs.graphext.com/api-docs/prepare/transform/slice
Extract a range/slice of elements from a column of texts or lists.
Using `start`, `stop` and `step` to define a range of indices, the corresponding range of elements is extracted
from each text or list in the input column.
Note: indices start at 0, and a `stop` of 3 means elements up to but *not* including the element at index 3
will be extracted. In particular this means simply specifying `"stop": 3` (setting or leaving `start` at its
default of 0), will extract 3 elements in total.
## Usage
The following examples show how the step can be used in a recipe.
Extract the first 100 characters from a text column
```stan theme={null}
slice(ds.description, {"start": 0, "stop": 100, "out_type": "text"}) -> (ds.description_short)
```
Extract the second element from a list column
```stan theme={null}
slice(ds.coordinates, {"start": 1, "stop": 2, "out_type": "number"}) -> (ds.latitude)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
slice(input: text|list, {
"param": value,
...
}) -> (output: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column containing texts or lists to extract a range of characters or elements from.
Contains the extracted slices. The type depends on the `out_type` parameter, and needs to be consistent with the transformation.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Index of the *first* element to be extracted from each string or list in the input column.
Index at which to stop including elements from the list.
The element *at* the `stop` index will *not* be included. As an example, `"start": 0, "stop": 3`
will include all elements up to but not including the element at index 3, thus extracting a
total of 3 elements.
Step size used to move from `start` to `stop` index.
E.g., if `"step": 2`, only every second element from the range \[`start`, `stop`] is returned.
Select types using their name.
Values must be one of the following:
`category` `date` `number` `boolean` `url` `sex` `text` `currency` `list[number]` `list[category]` `list[url]` `list[boolean]` `list[date]` `list[currency]`
# split_string
Source: https://docs.graphext.com/api-docs/prepare/transform/split_string
Split a single column containing texts into two.
The values of a text column will be split in two at the first occurrence of a given pattern, returning two new text columns.
For example, splitting a text column on the comma character (",") will produce two new columns: the first containing everything
before the first comma encountered in each text, and the second containing all text encountered after the comma.
If the specified split pattern was not encountered in any of the input texts, the first output column will contain
the original text, and the second column will contain missing values only (NaN).
## Usage
The following example shows how the step can be used in a recipe.
E.g. to split on the first comma encountered starting from the left of each text:
```stan theme={null}
split_string(ds.text, {"pattern": ","}) -> (ds.text_left, ds.text_right)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
split_string(input: text|category, {
"param": value,
...
}) -> (output_left: text, output_right: text)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A text column to split.
A text column containing the part to the left of the given split pattern.
A text column containing the part to the right of the given split pattern.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
A pattern of characters indicating where to split each text. By default uses the whitespace " ".
Whether to search for the pattern starting from the right instead of starting from the left (default).
# subtract
Source: https://docs.graphext.com/api-docs/prepare/transform/subtract
Subtract two or more numeric columns.
An additional constant may be used to subtract from the final result.
## Usage
The following examples show how the step can be used in a recipe.
To subtract column `number2` from column `number1`:
```stan theme={null}
subtract(ds.number1, ds.number2) -> (ds.difference)
```
To subtract the constant 1.23 from column `num_col`:
```stan theme={null}
subtract(ds.num_col, {"constant": 1.23}) -> (ds.difference)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
subtract(*columns: number, {
"param": value,
...
}) -> (result: number)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
One or numeric columns to subtract (in left-to-right order).
Numeric column containing the result of the subtraction.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
A constant to subtract from input data.
Whether to ignore NaNs or propagate them.
# sum
Source: https://docs.graphext.com/api-docs/prepare/transform/sum
Calculate the row-wise sum of numeric columns.
An additional constant may be used to add to the final result.
## Usage
The following examples show how the step can be used in a recipe.
To add columns `num1` and `num2`:
```stan theme={null}
sum(ds.num1, ds.num2) -> (ds.total)
```
To add the constant 3.141 to column `num1`:
```stan theme={null}
sum(ds.num1, {"constant": 3.141}) -> (ds.num1_plus_pi)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
sum(*columns: number, {
"param": value,
...
}) -> (result: number)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
One or numeric columns to sum.
Numeric column containing the result of the summation.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
A constant to add.
Whether to ignore NaNs or propagate them.
# time_interval
Source: https://docs.graphext.com/api-docs/prepare/transform/time_interval
Calculates the duration of a time interval between two dates (datetimes/timestamps).
The dates can be specified either as two datetime columns (in which case the second is subtracted from the first),
or as a single column and a reference date provided as a parameter (see `since` or `until` in parameters below).
If only one column is provided as input, one of `since` or `until` *must* be specified as a reference date.
## Usage
The following examples show how the step can be used in a recipe.
To get the positive number of hours since the last login of a user as of now:
```stan theme={null}
time_interval(ds.last_login, {
"unit": "hours",
"until": "now",
}) -> (ds.hours_elapsed)
```
Using `since` instead would lead to the same result but with negative hours:
```stan theme={null}
time_interval(ds.last_login, {
"unit": "hours",
"since": "now"
}) -> (ds.hours_ago)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
time_interval(date1: date, *date2: date, {
"param": value,
...
}) -> (interval: number)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column of datetimes.
An (optional) column of datetimes subtracted from the first.
The calculated interval in selected units.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Unit for duration.
The unit of measurement for the duration. Allowed values are: - "Y", "year" - "Q", "quarter" - "M", "month" - "W", "week" - "D", "day" - "h", "hour" - "m", "minute" - "s", "second" - "ms", "millisecond"
The unit name can be spelled in singular or plural and is case-insensitive.
Values must be one of the following:
`Y` `year` `Year` `years` `Years` `Q` `quarter` `Quarter` `quarters` `Quarters` `M` `month` `Month` `months` `Months` `W` `week` `Week` `weeks` `Weeks` `D` `day` `Day` `days` `Days` `h` `hour` `Hour` `hours` `Hours` `m` `minute` `Minute` `minutes` `Minutes` `s` `second` `Second` `seconds` `Seconds` `ms` `millisecond` `Millisecond` `milliseconds` `Milliseconds`
Return type of interval duration.
Whether the interval duration should always be returned as positive, independent of whether `date1` occurred before or after `date2`.
Date start reference for intervals.
If only one column is specified as input, a reference date relative to which the intervals will be calculated. The result will be `date1 - since`. I.e. intervals will be positive if dates in the column are more recent than the reference date (and negative otherwise). The date must be either a valid date string (preferrable month-first, e.g. "2021-12-31"), or the constant "now".
Date ending reference for intervals.
If only one column is specified as input, a reference date relative to which the intervals will be calculated. The result will be `until - date1`. I.e. intervals will be positive if the reference date is more recent than the dates in the column (and negative otherwise). The date must be either a valid date string (preferrable month-first, e.g. "2021-12-31"), or the constant "now".
# tokenize
Source: https://docs.graphext.com/api-docs/prepare/transform/tokenize
Parse texts and separate them into lists of tokens (words, lemmas, etc.).
## Usage
The following example shows how the step can be used in a recipe.
E.g. to convert text strings to lists of lower-cased words, ignoring any tokens that represent
punctuation (punct), URLs and stop words (stops), use:
```stan theme={null}
tokenize(ds.text, ds.lang, {
"tokens": {
"exclude": ["punct", "urls", "stops"]
}
}) -> (ds.tokens)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
tokenize(text: text, *lang: category, {
"param": value,
...
}) -> (tokens: list[category])
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column of texts to separate into tokens.
An (optional) column identifying the languages of the corresponding texts. It is used to identify the correct model (spaCy)
to use for each text. If the dataset doesn't contain such a column yet, it can be created using the `infer_language` step.
Ideally, languages should be expressed as two-letter
[ISO 639-1 language codes](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes), such as "en", "es" or "de" for
English, Spanish or German respectively. We also detect fully spelled out names such as "english", "German", "allemande"
etc., but it is not guaranteed that we will recognize all possible spellings correctly always, so ISO codes should be
preferred.
Alternatively, if all texts are in the same language, it can be identified with the `language` *parameter* instead.
A column of lists containing the tokens extracted from the texts.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Whether to enable support for additional languages.
By default, Arabic ("ar"), Catalan ("ca"), Basque ("eu"), and Turkish ("tu") are not enabled,
since they're supported only by a different class of language models (stanfordNLP's Stanza)
that is much slower than the rest. This parameter can be used to enable them.
Minimum number (or proportion) of texts to include a language in processing.
Any texts in a language with fewer documents than these will be ignored. Can be useful to speed up
processing when there is noise in the input languages, and when ignoring languages with a small number of
documents only is acceptable. Values smaller than 1 will be interpreted as a *proportion* of all texts, and
values greater than or equal to 1 as an *absolute number* of documents.
number.
Values must be in the following range:
```javascript theme={null}
0 < {_} < 1
```
integer.
Values must be in the following range:
```javascript theme={null}
1 ≤ {_} < inf
```
The language of inputs texts.
If all texts are in the same language, it can be specified here instead of passing it as an input column. The language will be used to identify the correct spaCy model to parse and analyze the texts. For allowed values, see the comment regarding the `lang` column above.
Configure how tokens are extracted and represented in the output.
Define the kinds of tokens to extract, how to represent them, and their minimum or maximum frequency in the
dataset to be included in the result.
Representation of the individual tokens to extract.
I.e. whether verbatim (text/ortho), lower(-case) or lemmatized. Also see spaCy's attribute reference
in [this table](https://spacy.io/api/token#attributes) for further information.
Values must be one of the following:
* `orth`
* `lemma`
* `lower`
* `text`
Which kinds of tokens to exclude.
Valid filters are stop words (`stops`), URLs (`urls`), punctuation (`punct`), digits,
tokens containing non-alphabetic characters (`non_alpha`), and tokens containing non-ascii
characters (`non_ascii`).
Each item in array.
Values must be one of the following:
`stops` `urls` `punct` `digits` `non_alpha` `non_ascii`
Token frequency filter.
Filters tokens based on the number of texts they occur in.
Minimum number of rows.
Tokens not occurring in at least these many rows (texts) will be excluded.
Values must be in the following range:
```javascript theme={null}
0 ≤ min_rows < inf
```
Maximum proportion of rows.
Tokens occurring in more than this proportion of rows (texts) will be excluded.
Values must be in the following range:
```javascript theme={null}
0 ≤ max_rows ≤ 1
```
Keep n most frequent tokens.
Whether to always include the n most frequent tokens, independent of the other filter parameters.
Set to `null` to ignore.
Values must be in the following range:
```javascript theme={null}
0 ≤ keep_top_n < inf
```
Exclude n most frequent tokens.
Even if they passed the other filter conditions.
Values must be in the following range:
```javascript theme={null}
0 ≤ filter_top_n < inf
```
Filter per language.
Apply filter conditions separately to texts grouped by language, rather than across all texts.
# trim_frequencies
Source: https://docs.graphext.com/api-docs/prepare/transform/trim_frequencies
Remove values whose frequencies (counts) are above/below a given threshold.
Affected categories are replaced with the missing value (NaN).
## Usage
The following examples show how the step can be used in a recipe.
To remove categories ocurring fewer than 2 times in the column `cat_col`:
```stan theme={null}
trim_frequencies(ds.cat_col, {"freq_min": 2}) -> (ds.cat_trimmed)
```
To only keep the 10 most frequent categories in column `cat_col`:
```stan theme={null}
trim_frequencies(ds.cat_col, {"n_most_common": 10}) -> (ds.cat_top10)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
trim_frequencies(input: category|list[category], {
"param": value,
...
}) -> (output: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A categorical column to trim.
A categorical column with fewer categories than the input.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
The number N indicating how many of the most common values to filter (in descending order).
Values must be in the following range:
```javascript theme={null}
0 ≤ n_most_common < inf
```
Values with a lower frequency (count) than this will be removed.
Values must be in the following range:
```javascript theme={null}
1 ≤ freq_min < inf
```
Values with a higher frequency (count) than this will be removed.
Values must be in the following range:
```javascript theme={null}
1 ≤ freq_max < inf
```
# unique
Source: https://docs.graphext.com/api-docs/prepare/transform/unique
Extracts the unique elements in each list/array.
Effectively deduplicates the input lists.
## Usage
The following example shows how the step can be used in a recipe.
This step has no parameters, so it's simply:
```stan theme={null}
unique(ds.input_lists) -> (ds.unique_elems)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
unique(input: list) -> (output: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A column containing lists.
A column of lists of the same type as the input, containing only the unique elements from each input lists.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
This step doesn't expect any configuration.
# unpack_list
Source: https://docs.graphext.com/api-docs/prepare/transform/unpack_list
Unpack (extract) items from a column of lists into separate columns.
The first output column will contain the items in the first position of the input lists, the second column items in the second position etc.
## Usage
The following examples show how the step can be used in a recipe.
Extract the first 3 items from a list column into separate columns
```stan theme={null}
unpack_list(ds.coordinates, {"n_items": 3}) => (ds.coord_parts)
```
Extract items at positions 1 and 2 with a custom prefix
```stan theme={null}
unpack_list(ds.names, {"start": 1, "n_items": 2, "prefix": "name"}) => (ds.name_parts)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
unpack_list(lists: list, {
"param": value,
...
}) -> (list_items: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
An input column containing lists to unpack.
The output dataset containing list elements in individual columns.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
List index of first element to extract.
Values must be in the following range:
```javascript theme={null}
0 ≤ start < inf
```
Total number of consecutive items to extract from lists.
Values must be in the following range:
```javascript theme={null}
1 ≤ n_items < inf
```
Prefix for names of generated columns.
By default, will use the output dataset's name concatenated with "\_0", "\_1" etc. for the first
extracted column, the second column etc. respectively. I.e. you name the output dataset of this step
`list_items`, then its columns will be named "list\_items\_0", "list\_items\_1" etc. If a prefix is provided,
this will be used *instead* of the output dataset's name.
A list of names for the columns in the output dataset.
Will be used only if the number of names passed matches the `n_items` parameter.
The name of a single column in the ouput.
# configure_category_colors
Source: https://docs.graphext.com/api-docs/report/configure_ui/configure_category_colors
Configures the color of the categories of a categorical or text column.
Maps specific category values to hex colors. Each key in the params object is a category name and each value is a hex color code (e.g. "#FF0000"). Categories not explicitly mapped will use the default color palette.
This is a UI configuration step that affects how the project is displayed in Graphext. It applies to the dataset referenced in its inputs. If your recipe produces multiple datasets (e.g. a filtered dataset that is then passed to create\_project alongside the original), you need to add separate configure steps for each dataset you want to configure.
## Usage
The following example shows how the step can be used in a recipe.
Change the colors of the category 'positive' to white, 'negative' to black and 'neutral' to green
```stan theme={null}
configure_category_colors(ds.sentiment, {
"postive": "#FFFFFF",
"negative": "#000000",
"neutral": "#00ff00"
})
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
configure_category_colors(column: category|list[category]|text|boolean|list[boolean], {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Column to be configured.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Category colors.
A mapping of category names to colors.
Values must match the following regex pattern:
```regex theme={null}
^#([a-fA-F0-9]{6}|[a-fA-F0-9]{3})$
```
# configure_category_labels
Source: https://docs.graphext.com/api-docs/report/configure_ui/configure_category_labels
Configures the labels generated for each category.
Note that for each possible `kind` parameter (`"significantTerms"`, `"sequential"`, or `"originals"`), this step has different configuration options. See subsections below for more details.
## Usage
The following examples show how the step can be used in a recipe.
```stan theme={null}
configure_category_labels(ds.cluster, { "kind": "originals" })
```
E.g., to show sequentially numbered labels with the "cluster" prefix (cluster-1, cluster-2 etc.), use:
```stan theme={null}
configure_category_labels(ds.cluster, { "kind": "sequential", "prefix": "cluster-" })
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
configure_category_labels(column: category, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Column to be configured.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Label categories with significant terms.
Significant terms column.
This column's significant terms will be shown as category labels.
In what order to show the categories.
Values must be one of the following:
* `BACKGROUND`
* `FOREGROUND`
* `UPLIFT`
* `TFIDF`
* `ORDINAL`
Label categories sequentially (1..N).
String that will be placed at the beginning of each label.
Use column's original labels.
# configure_category_order
Source: https://docs.graphext.com/api-docs/report/configure_ui/configure_category_order
Configures the order of categories in a categorical or list of categories column.
Controls the display order of categories in charts and filters. The ordering method is specified by the `kind` parameter: `"BACKGROUND"` orders by total row count, `"FOREGROUND"` by count in the current selection, `"UPLIFT"` by over-representation in the selection, `"TFIDF"` by term frequency-inverse document frequency, and `"ORDINAL"` preserves a natural order.
This is a UI configuration step that affects how the project is displayed in Graphext. It applies to the dataset referenced in its inputs. If your recipe produces multiple datasets (e.g. a filtered dataset that is then passed to create\_project alongside the original), you need to add separate configure steps for each dataset you want to configure.
## Usage
The following example shows how the step can be used in a recipe.
```stan theme={null}
configure_category_order(ds.cluster, { "kind": "BACKGROUND" })
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
configure_category_order(column: category|list[category]|text, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Categorical column to be configured (may be a column of lists of categories).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Kind of ordering.
Determines the order of categories. E.g. to show first the categories having the greatest row count in the whole dataset,
select `"BACKGROUND"`. To show those categories first having the greatest number of rows in the current selection, use
`"FOREGROUND"`, etc.
Values must be one of the following:
* `BACKGROUND`
* `FOREGROUND`
* `UPLIFT`
* `TFIDF`
* `ORDINAL`
# configure_color_palette
Source: https://docs.graphext.com/api-docs/report/configure_ui/configure_color_palette
Configures the base global color palettes to use when coloring categorical or quantitative columns.
Sets the global color palettes used across the project for different column types. You can configure separate palettes for categorical columns, sequential single-hue (numeric with one direction), sequential multi-hue (numeric with gradient), and diverging (numeric with a center point). If not configured, Graphext uses its default palettes.
This is a UI configuration step that affects how the project is displayed in Graphext. It applies to the dataset referenced in its inputs. If your recipe produces multiple datasets (e.g. a filtered dataset that is then passed to create\_project alongside the original), you need to add separate configure steps for each dataset you want to configure.
## Usage
The following example shows how the step can be used in a recipe.
Use Osiris, Greens, Magma & Red-Yellow-Green palettes.
```stan theme={null}
configure_color_palette({
"categorical": "Osiris",
"sequentialSingleHue": "Greens",
"sequentialMultiHue": "Magma",
"diverging": "Red-Yellow-Green"
})
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
configure_color_palette(, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Categorical palette.
Values must be one of the following:
* `Horus`
* `Osiris`
* `Re`
Sequential Single-Hue palette.
Values must be one of the following:
* `Blues`
* `Greens`
* `Oranges`
* `Purples`
* `Reds`
Sequential Multi-Hue palette.
Values must be one of the following:
* `Viridis`
* `Magma`
* `Plasma`
Diverging palette.
Values must be one of the following:
* `Red-Blue`
* `Red-Yellow-Green`
# configure_column_metadata
Source: https://docs.graphext.com/api-docs/report/configure_ui/configure_column_metadata
Configures the label and/or description of a column.
Sets a human-readable label, description, and/or number format for a column. The `label` replaces the column name in the UI. The `description` appears as a tooltip. The `format` uses d3-format syntax for number display (e.g. `",.2f"` for two decimal places with thousands separator).
This is a UI configuration step that affects how the project is displayed in Graphext. It applies to the dataset referenced in its inputs. If your recipe produces multiple datasets (e.g. a filtered dataset that is then passed to create\_project alongside the original), you need to add separate configure steps for each dataset you want to configure.
## Usage
The following example shows how the step can be used in a recipe.
```stan theme={null}
configure_column_metadata(ds.avatar, { "label": "Photo", "description": "Profile image of a user" })
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
configure_column_metadata(column: column, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Column to be configured.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Label of the column.
Description of the column.
Format the column's numbers using d3-format (e.g., ',.3f' will format the number with 3 decimal places).
# configure_column_view_modes
Source: https://docs.graphext.com/api-docs/report/configure_ui/configure_column_view_modes
Configures the visualization mode for columns in the filters panel.
Controls how each column is rendered in the filters panel. Each key is a column name and each value is a view mode. Available modes depend on the column type but include options like `"barChart"`, `"list"`, `"histogram"`, `"scatterPlot"`, etc.
This is a UI configuration step that affects how the project is displayed in Graphext. It applies to the dataset referenced in its inputs. If your recipe produces multiple datasets (e.g. a filtered dataset that is then passed to create\_project alongside the original), you need to add separate configure steps for each dataset you want to configure.
## Usage
The following example shows how the step can be used in a recipe.
Sets the age column as list and the vote\_intention column as barChart
```stan theme={null}
configure_column_view_modes({ "age": "list", "vote_intention": "barChart" })
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
configure_column_view_modes(, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
One or more additional parameters.
Values must be one of the following:
* `barChart`
* `list`
# configure_column_visibility
Source: https://docs.graphext.com/api-docs/report/configure_ui/configure_column_visibility
Configures the visibility of a column in different Graphext sections.
Controls whether a column is visible, pinned (always shown), or hidden in the Graph and Details sections of the project. Pinned columns appear prominently in filters. Hidden columns are still available in the dataset but not shown by default.
This is a UI configuration step that affects how the project is displayed in Graphext. It applies to the dataset referenced in its inputs. If your recipe produces multiple datasets (e.g. a filtered dataset that is then passed to create\_project alongside the original), you need to add separate configure steps for each dataset you want to configure.
## Usage
The following example shows how the step can be used in a recipe.
```stan theme={null}
configure_column_visibility(ds.bio, { "graph": "pinned", "details": "hidden" })
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
configure_column_visibility(column: column, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Column to be configured.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
How to show a column in the Graph section.
Values must be one of the following:
* `pinned`
* `hidden`
How to show a column in the Details section.
Values must be one of the following:
* `pinned`
* `hidden`
# configure_columns_order
Source: https://docs.graphext.com/api-docs/report/configure_ui/configure_columns_order
Configures the order of columns (filters) in the Graph and Details sections.
By default the order is identical to that in the input dataset.
If only a subset of columns is specified here, the rest will maintain the original order.
## Usage
The following examples show how the step can be used in a recipe.
First cross filters: bio, age and deparment
```stan theme={null}
configure_columns_order(ds.bio, ds.salary, ds.age, ds.department)
```
First columns in data table: bio, age and deparment
```stan theme={null}
configure_columns_order(ds.bio, ds.salary, ds.age, ds.department, { "for": "dataTable" })
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
configure_columns_order(*columns: column, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Given order applies to data table or cross filter (cross filter when none).
Values must be one of the following:
* `dataTable`
* `crossFilter`
# configure_dataset_metadata
Source: https://docs.graphext.com/api-docs/report/configure_ui/configure_dataset_metadata
Configures the info_source, label and/or description of a dataset.
Sets a label, description, and/or source URL for the dataset itself. The `label` is the display name of the dataset in the UI. The `description` provides context about the data. The `info_source` is a URL linking to the original data source.
This is a UI configuration step that affects how the project is displayed in Graphext. It applies to the dataset referenced in its inputs. If your recipe produces multiple datasets (e.g. a filtered dataset that is then passed to create\_project alongside the original), you need to add separate configure steps for each dataset you want to configure.
## Usage
The following example shows how the step can be used in a recipe.
```stan theme={null}
configure_dataset_metadata(ds, { "label": "Dataset label", "description": "Useful dataset description", "info_source": "https://www.example.com" })
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
configure_dataset_metadata(dataset: dataset, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Dataset to be configured.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Source URL.
Values must match the following regex pattern:
```regex theme={null}
^https?://
```
Description of the dataset.
Label of the dataset.
# configure_detail_view
Source: https://docs.graphext.com/api-docs/report/configure_ui/configure_detail_view
Select the preferred columns to customize a row detail view.
Defines which columns appear when a user clicks on a node to see its detail view. Only the specified columns will be shown, in the order provided. If not configured, all visible columns are shown.
This is a UI configuration step that affects how the project is displayed in Graphext. It applies to the dataset referenced in its inputs. If your recipe produces multiple datasets (e.g. a filtered dataset that is then passed to create\_project alongside the original), you need to add separate configure steps for each dataset you want to configure.
## Usage
The following example shows how the step can be used in a recipe.
It will show the detail view with column A, column B and column C values.
```stan theme={null}
configure_detail_view(ds.columnA, ds.columnB, ds.columnC)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
configure_detail_view(*columns: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
This step doesn't expect any configuration.
# configure_discarded_categories
Source: https://docs.graphext.com/api-docs/report/configure_ui/configure_discarded_categories
Configures a minimum number of rows in a category below which the category will be hidden from the variable's filter view.
Hides categories with a low number of rows from the filter panel. The `thresholds` parameter defines the minimum count or percentage below which a category is hidden. This is useful for decluttering filters when there are many infrequent categories.
This is a UI configuration step that affects how the project is displayed in Graphext. It applies to the dataset referenced in its inputs. If your recipe produces multiple datasets (e.g. a filtered dataset that is then passed to create\_project alongside the original), you need to add separate configure steps for each dataset you want to configure.
## Usage
The following example shows how the step can be used in a recipe.
```stan theme={null}
configure_discarded_categories(ds.cluster, { "thresholds": [{ "target": "EVERYTHING", "reference": "PERCENTAGE", "value": 20 }] })
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
configure_discarded_categories(column: category|list[category]|text, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
The column to configure.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
A list of threshold configurations.
A categorical column can have two kinds of thresholds determining whether specific categories will be
hidden from its view in the UI: a minimum number of rows in the current *selection* below which a category
will be hidden, or a minimum number of rows in the *whole dataset* (*everything*).
The `thresholds` parameter should be a list containing 1 or 2 objects: the configuration of a *selection*
threshold, and/or the configuration of a threshold for *everything*.
array.
Configure categories to be discarded (hidden) in terms of their occurrence in the *whole dataset*.
Categories with a number (or percentage) of rows in the *whole dataset* less than `value` will be discarded (hidden from the variable's filter view).
Whether to apply the threshold to the current selection of rows or all rows in the dataset.
Whether to interpret the threshold value as an absolute (count) or percentage of rows.
Values must be one of the following:
* `ABSOLUTE`
* `PERCENTAGE`
Categories less frequent than this value will be discarded (hidden).
array.
Configure categories to be discarded (hidden) in terms of their occurrence in the *current selection*.
Categories with a number (or percentage) of rows in the current selection less than `value` will be discarded (hidden from the variable's filter view).
Whether to apply the threshold to the current selection of rows or all rows in the dataset.
Whether to interpret the threshold value as an absolute (count) or percentage of rows.
Values must be one of the following:
* `ABSOLUTE`
* `PERCENTAGE`
Categories less frequent than this value will be discarded (hidden).
array.
Configure categories to be discarded (hidden) in terms of their occurrence in the *whole dataset*.
Categories with a number (or percentage) of rows in the *whole dataset* less than `value` will be discarded (hidden from the variable's filter view).
Whether to apply the threshold to the current selection of rows or all rows in the dataset.
Whether to interpret the threshold value as an absolute (count) or percentage of rows.
Values must be one of the following:
* `ABSOLUTE`
* `PERCENTAGE`
Categories less frequent than this value will be discarded (hidden).
Configure categories to be discarded (hidden) in terms of their occurrence in the *current selection*.
Categories with a number (or percentage) of rows in the current selection less than `value` will be discarded (hidden from the variable's filter view).
Whether to apply the threshold to the current selection of rows or all rows in the dataset.
Whether to interpret the threshold value as an absolute (count) or percentage of rows.
Values must be one of the following:
* `ABSOLUTE`
* `PERCENTAGE`
Categories less frequent than this value will be discarded (hidden).
array.
Configure categories to be discarded (hidden) in terms of their occurrence in the *current selection*.
Categories with a number (or percentage) of rows in the current selection less than `value` will be discarded (hidden from the variable's filter view).
Whether to apply the threshold to the current selection of rows or all rows in the dataset.
Whether to interpret the threshold value as an absolute (count) or percentage of rows.
Values must be one of the following:
* `ABSOLUTE`
* `PERCENTAGE`
Categories less frequent than this value will be discarded (hidden).
Configure categories to be discarded (hidden) in terms of their occurrence in the *whole dataset*.
Categories with a number (or percentage) of rows in the *whole dataset* less than `value` will be discarded (hidden from the variable's filter view).
Whether to apply the threshold to the current selection of rows or all rows in the dataset.
Whether to interpret the threshold value as an absolute (count) or percentage of rows.
Values must be one of the following:
* `ABSOLUTE`
* `PERCENTAGE`
Categories less frequent than this value will be discarded (hidden).
# configure_graph_layout
Source: https://docs.graphext.com/api-docs/report/configure_ui/configure_graph_layout
Configures the x & y columns used to map node positions in the graph.
Specifies which two numeric columns contain the x and y coordinates for positioning nodes in the graph view. Typically these are the output of a layout step like `layout_network` or `layout_coordinates`.
This is a UI configuration step that affects how the project is displayed in Graphext. It applies to the dataset referenced in its inputs. If your recipe produces multiple datasets (e.g. a filtered dataset that is then passed to create\_project alongside the original), you need to add separate configure steps for each dataset you want to configure.
## Usage
The following example shows how the step can be used in a recipe.
```stan theme={null}
configure_graph_layout(ds.x, ds.y)
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
configure_graph_layout(x: number, y: number
)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Numerical column with the x position in the graph.
Numerical column with the y position in the graph.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
This step doesn't expect any configuration.
# configure_graph_regions
Source: https://docs.graphext.com/api-docs/report/configure_ui/configure_graph_regions
Configures the column that is displayed as the label of the graph region.
Configures the column that is displayed as the label of the graph region
A good choice would be a column containing a short text or label identifying or summarizing the corresponding group of rows in the dataset in a meaningful way, such as the cluster or segmentation.
## Usage
The following example shows how the step can be used in a recipe.
```stan theme={null}
configure_graph_regions(ds.field, { margin: 32 })
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
configure_graph_regions(field: column, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Sets the space between labels.
# configure_metrics
Source: https://docs.graphext.com/api-docs/report/configure_ui/configure_metrics
Configures the metrics to be calculated and displayed.
This step allows you to define custom metrics that can be calculated from the dataset.
Each metric is defined by a `name`, an optional `description`, and a `script` that calculates the metric. Optionally, you can set a `hidden` property to true to hide the metric from the UI.
Inside one metric, you can use another metric through the available object `Metric` like this: `Metric["licenses_to_renew"]`.
The most important part of the metric definition is the Javascript code. This script can use ES2023 JS syntax and it must implement a function body that returns a number representing the computed metric value. Within a script, the following functions and objects are available:
* `ds` and `dsAll`: Accessors to the dataset used as input. `dsAll` refers to the entire dataset, while `ds` is affected by the applied selection. If no selection is made, both accessors refer to the same dataset.
* `count(ds)`: A function that returns the number of rows in the dataset.
* `where("queryString", () => { /* 'ds' accessor is a subset filtered by the query and the global selection */ })`: A function that applies a filter to the `ds` accessor. Note that `dsAll` remains unaffected. Multiple `where` calls can be nested. See [Advanced query filters](https://docs.graphext.com/concepts/ds-concepts/filter-queries#advanced-filter-queries) for more information on query strings.
* `Metrics`: You can reference another metrics within a metrics script using the `Metrics` object, e.g., `Metrics["licenses_to_renew"]`.
Each column in the dataset has several aggregation operations available, depending on its data type. These operations can be accessed as if they were fields of the column, using dot notation. For example:
* For a numerical column 'sales': ds.sales.sum or ds.sales.mean
* For a categorical column 'department': ds.department.uniqueValues
* For a text column 'description': ds.description.wordCount
* For a date column 'order\_date': ds.order\_date.min or ds.order\_date.max
You can use the following operations directly in your metric script to perform calculations on the dataset columns:
ALL ColumnTypes:
* `.nullRows`: Returns the number of null rows in the column.
* `.validRows`: Returns the number of non-null rows in the column.
NUMERICAL and DATE:
* `.count`: Returns the number of non-null values in the column. If the column is a List this will count each element.
* `.sum`: Calculates the sum of all values in the column.
* `.stddev`: Calculates the standard deviation of the values in the column.
* `.variance`: Calculates the variance of the values in the column.
* `.mean`: Calculates the average of all values in the column.
* `.min`: Returns the minimum value in the column.
* `.p25`: Calculates the 25th percentile of the values in the column.
* `.p50`: Calculates the 50th percentile (median) of the values in the column.
* `.median`: An alias for `.p50`, calculates the median of the values in the column.
* `.p75`: Calculates the 75th percentile of the values in the column.
* `.max`: Returns the maximum value in the column.
CATEGORICAL:
* `.count`: Returns the number of non-null values in the column. If the column is a List this will count each element.
* `.uniqueValues`: Returns the number of unique values in the column.
TEXT:
* `.wordCount`: Calculates the total number of words across all non-null rows in the column.
## Usage
The following examples show how the step can be used in a recipe.
Calculate the average order value on the selected rows
```stan theme={null}
configure_metrics(ds, {
"metrics": [
{
"name": "average_order_value",
"description": "Average order value",
"script": "return ds.sales.sum / ds.order_id.uniqueValues;"
}
]
})
```
Calculate the percentage of total sales in the selected rows vs the total sales in the dataset
```stan theme={null}
configure_metrics(ds, {
"metrics": [
{
"name": "sales_percentage",
"script": "return 100 * ds.sales.sum / dsAll.sales.sum;"
}
]
})
```
Calculate the percentage of clients older than 65 years old relative to the selection.
```stan theme={null}
configure_metrics(ds, {
"metrics": [
{
"name": "percentage_older_than_65",
"script": "return 100 * where('age: >65', () => { return count(ds); }) / count(ds);"
}
]
})
```
Metrics references. Normalize the sum of accidents per population assuming each row is a country.
```stan theme={null}
configure_metrics(ds, {
"metrics": [
{
"name": "accidents_sum",
"description": "Sum of accidents in all countries",
"script": "return ds.accidents.sum;"
},
{
"name": "accidents_per_population",
"description": "Accidents normalized per population",
"script": "return Metrics['accidents_sum'] / ds.population.sum;"
}
]
})
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
configure_metrics(ds: dataset, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Dataset to be used for calculating metrics.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Metrics list.
The list of metrics to be calculated and displayed has the following properties:
* name: The ID for the metric. No duplicated names are allowed.
* description: An optional description of the metric.
* script: The javascript code that calculates the metric. The metric should return a number.
* hidden: A boolean that indicates whether the metric should be hidden. Useful for auxiliary metrics.
Metric name.
An ID for the metric. No duplicated names are allowed.
Metric description.
An optional description of the metric.
Metric script.
Hidden metric.
A boolean that indicates whether the metric should be hidden. Useful for auxiliary metrics.
Show difference.
Whether to show the difference between the metric of the selection and the metric of the total population.
This can be configured individually for each metric. If not specified, the global configuration will be used.
Values must be one of the following:
* `ABSOLUTE`
* `RELATIVE`
unit.
Unit symbol.
The symbol of the unit (e.g. '\$', 'kg').
Unit display position.
The position of the unit symbol in the metric display.
Values must be one of the following:
* `suffix`
* `prefix`
Show difference.
Whether to show the difference between the metric of the selection and the metric of the total population.
Values must be one of the following:
* `ABSOLUTE`
* `RELATIVE`
# configure_node_color
Source: https://docs.graphext.com/api-docs/report/configure_ui/configure_node_color
Configures the column that is used for coloring the nodes by default.
Sets the default column used to color nodes in the graph view. Any column type works: categorical columns assign discrete colors per category, numeric columns use a gradient, and date columns use a temporal gradient. Users can change the coloring column interactively, but this sets the initial default.
This is a UI configuration step that affects how the project is displayed in Graphext. It applies to the dataset referenced in its inputs. If your recipe produces multiple datasets (e.g. a filtered dataset that is then passed to create\_project alongside the original), you need to add separate configure steps for each dataset you want to configure.
## Usage
The following shows how the step can be used in a recipe.
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
configure_node_color(color: column)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Column to use as color.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
This step doesn't expect any configuration.
# configure_node_connections
Source: https://docs.graphext.com/api-docs/report/configure_ui/configure_node_connections
Configures how the connections between the nodes are visualized.
Controls how network links (edges) between nodes are displayed. The `visible` parameter shows or hides the connection lines, and `directed` controls whether arrows indicate direction from source to target.
This is a UI configuration step that affects how the project is displayed in Graphext. It applies to the dataset referenced in its inputs. If your recipe produces multiple datasets (e.g. a filtered dataset that is then passed to create\_project alongside the original), you need to add separate configure steps for each dataset you want to configure.
## Usage
The following examples show how the step can be used in a recipe.
Show connections and represent the direction from source to target
```stan theme={null}
configure_node_connections({"visible": true, "directed": true})
```
Hide connections and show only the nodes
```stan theme={null}
configure_node_connections({"visible": false, "directed": false})
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
configure_node_connections(*links: list[number], {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Links are shown.
Links are directed.
# configure_node_picture
Source: https://docs.graphext.com/api-docs/report/configure_ui/configure_node_picture
Configures the pictures associated with the nodes of the network.
Configures the icon drawn on each node of the network using a column of URLs pointing to downloadable image files.
## Usage
The following shows how the step can be used in a recipe.
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
configure_node_picture(pics: url)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Column of URLs pointing to .jpeg or .png files to be used as picture icons of the nodes.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
This step doesn't expect any configuration.
# configure_node_size
Source: https://docs.graphext.com/api-docs/report/configure_ui/configure_node_size
Configures the column, the minimum and the maximum that are used for sizing the nodes by default.
Note that larger nodes are more likely to have their title drawn in the network view.
## Usage
The following shows how the step can be used in a recipe.
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
configure_node_size(*relevance: number|date, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Sets the minimum value of the node size.
Values must be in the following range:
```javascript theme={null}
0.01 ≤ min ≤ 1.5
```
Sets the maximum value of the node size.
Values must be in the following range:
```javascript theme={null}
0.01 ≤ max ≤ 1.5
```
# configure_node_title
Source: https://docs.graphext.com/api-docs/report/configure_ui/configure_node_title
Configures the column that is displayed as the title of the node.
The title identifies each node visually and can be used to filter the network.
A good choice would be a column containing a short text or label identifying or summarizing the corresponding row in the dataset in a meaningful way, such as the title of an article or the ID of a customer.
Note that larger nodes are more likely to have their label drawn in the network view. See *configure\_node\_size*.
## Usage
The following example shows how the step can be used in a recipe.
```stan theme={null}
configure_node_title(ds.title, { margin: 32 })
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
configure_node_title(title: column, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Sets the space between titles.
# configure_node_url
Source: https://docs.graphext.com/api-docs/report/configure_ui/configure_node_url
Configures the urls associated with the nodes of the network.
Right-clicking on a node will open a new browser tab and load the associated URL, if one is present.
## Usage
The following shows how the step can be used in a recipe.
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
configure_node_url(url: url)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
URL Column used to associate a link with each node, which opens upon right-click.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
This step doesn't expect any configuration.
# configure_rows_order
Source: https://docs.graphext.com/api-docs/report/configure_ui/configure_rows_order
Configures the order of the rows in the Data section.
By default the order is identical to that in the input dataset, and row indices will be used to decide any ties found.
If you're selecting a list-like column, only the highest (ascending) or lowest (descending) value of the column will be used for sorting.
## Usage
The following examples show how the step can be used in a recipe.
Order table rows from highest to lowest salary
```stan theme={null}
configure_rows_order(ds.salary, { "order": "ascending" })
```
Order table rows from lowest to highest age
```stan theme={null}
configure_rows_order(ds.age, { "order": "descending" })
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
configure_rows_order(column_to_order: column, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Column to be used to order the rows.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Sort direction to be applied to data table rows order.
Values must be one of the following:
* `ascending`
* `descending`
# configure_sections
Source: https://docs.graphext.com/api-docs/report/configure_ui/configure_sections
Configures pinned Graphext sections.
Controls which sections of the Graphext interface are pinned (always visible) in the project. Available sections include `compare`, `correlations`, `dashboards`, `graph`, `insights`, `models`, and `plot`. Pinned sections appear expanded by default.
This is a UI configuration step that affects how the project is displayed in Graphext. It applies to the dataset referenced in its inputs. If your recipe produces multiple datasets (e.g. a filtered dataset that is then passed to create\_project alongside the original), you need to add separate configure steps for each dataset you want to configure.
## Usage
The following example shows how the step can be used in a recipe.
```stan theme={null}
configure_sections({ compare: { pinned: true }, correlations: { pinned: true } })
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
configure_sections(, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Configuration for Graphext sections.
Configuration for Graphext sections.
Defines if the section is pinned to the top bar.
Configuration for Graphext sections.
Configuration for Graphext sections.
Defines if the section is pinned to the top bar.
Configuration for Graphext sections.
Configuration for Graphext sections.
Defines if the section is pinned to the top bar.
Configuration for Graphext sections.
Configuration for Graphext sections.
Defines if the section is pinned to the top bar.
Configuration for Graphext sections.
Configuration for Graphext sections.
Defines if the section is pinned to the top bar.
Configuration for Graphext sections.
Configuration for Graphext sections.
Defines if the section is pinned to the top bar.
Configuration for Graphext sections.
Configuration for Graphext sections.
Defines if the section is pinned to the top bar.
# configure_tagged_columns
Source: https://docs.graphext.com/api-docs/report/configure_ui/configure_tagged_columns
Create groups of variables by tagging the provided variable(s) with the specified tag.
## Usage
The following example shows how the step can be used in a recipe.
Tag columns with tags
```stan theme={null}
configure_tagged_columns(ds, {"very important" : ["columna"], "other tag": ["columna", "columnb"}})`
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
configure_tagged_columns(ds: dataset, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
One or more additional parameters.
Each item in array.
# Configure UI
Source: https://docs.graphext.com/api-docs/report/configure_ui/index
| Step | Fast | Description |
| ------------------------------------------------------------------------------------------------ | ---- | ----------------------------------------------------------------------------------------------------------------------- |
| [configure\_category\_colors](/api-docs/report/configure_ui/configure_category_colors) | | Configures the color of the categories of a categorical or text column |
| [configure\_category\_labels](/api-docs/report/configure_ui/configure_category_labels) | | Configures the labels generated for each category |
| [configure\_category\_order](/api-docs/report/configure_ui/configure_category_order) | | Configures the order of categories in a categorical or list of categories column |
| [configure\_color\_palette](/api-docs/report/configure_ui/configure_color_palette) | | Configures the base global color palettes to use when coloring categorical or quantitative columns |
| [configure\_column\_metadata](/api-docs/report/configure_ui/configure_column_metadata) | | Configures the label and/or description of a column |
| [configure\_column\_view\_modes](/api-docs/report/configure_ui/configure_column_view_modes) | | Configures the visualization mode for columns in the filters panel |
| [configure\_column\_visibility](/api-docs/report/configure_ui/configure_column_visibility) | | Configures the visibility of a column in different Graphext sections |
| [configure\_columns\_order](/api-docs/report/configure_ui/configure_columns_order) | | Configures the order of columns (filters) in the Graph and Details sections |
| [configure\_dataset\_metadata](/api-docs/report/configure_ui/configure_dataset_metadata) | | Configures the info\_source, label and/or description of a dataset |
| [configure\_detail\_view](/api-docs/report/configure_ui/configure_detail_view) | | Select the preferred columns to customize a row detail view |
| [configure\_discarded\_categories](/api-docs/report/configure_ui/configure_discarded_categories) | | Configures a minimum number of rows in a category below which the category will be hidden from the variable's filter v… |
| [configure\_graph\_layout](/api-docs/report/configure_ui/configure_graph_layout) | | Configures the x & y columns used to map node positions in the graph |
| [configure\_graph\_regions](/api-docs/report/configure_ui/configure_graph_regions) | | Configures the column that is displayed as the label of the graph region |
| [configure\_metrics](/api-docs/report/configure_ui/configure_metrics) | | Configures the metrics to be calculated and displayed |
| [configure\_node\_color](/api-docs/report/configure_ui/configure_node_color) | | Configures the column that is used for coloring the nodes by default |
| [configure\_node\_connections](/api-docs/report/configure_ui/configure_node_connections) | | Configures how the connections between the nodes are visualized |
| [configure\_node\_picture](/api-docs/report/configure_ui/configure_node_picture) | | Configures the pictures associated with the nodes of the network |
| [configure\_node\_size](/api-docs/report/configure_ui/configure_node_size) | | Configures the column, the minimum and the maximum that are used for sizing the nodes by default |
| [configure\_node\_title](/api-docs/report/configure_ui/configure_node_title) | | Configures the column that is displayed as the title of the node |
| [configure\_node\_url](/api-docs/report/configure_ui/configure_node_url) | | Configures the urls associated with the nodes of the network |
| [configure\_rows\_order](/api-docs/report/configure_ui/configure_rows_order) | | Configures the order of the rows in the Data section |
| [configure\_sections](/api-docs/report/configure_ui/configure_sections) | | Configures pinned Graphext sections |
| [configure\_tagged\_columns](/api-docs/report/configure_ui/configure_tagged_columns) | | Create groups of variables by tagging the provided variable(s) with the specified tag |
# create_compare_insight
Source: https://docs.graphext.com/api-docs/report/create_insight/create_compare_insight
Create a new insight from the Compare section.
## Usage
The following shows how the step can be used in a recipe.
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
create_compare_insight(, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Title of the insight.
Columns from which the insight is created.
Each item in array.
Columns that will not be used to create the insight.
Each item in array.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Sets the correlation graph as expanded.
If true, the insight will contain an expanded chart in order to show the maximum number of categories possible.
Useful to choose the ordering kind of the columns, by difference or by similarity.
Values must be one of the following:
* `differentFirst`
* `similarFirst`
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
Sets the visualization type.
In relative and absolute modes the comparing segments will be rendered as charts, but in tabular mode a table is shown.
Values must be one of the following:
* `relative`
* `absolute`
* `tabular`
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
Element visibility.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript theme={null}
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript theme={null}
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript theme={null}
0 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript theme={null}
0 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
Array of segments to use in the insight.
Defines some behavior in the UI for this comparing segment.
Values must be one of the following:
* `VALUES`
* `RANGE`
* `EVERYTHING`
* `SELECTION`
Specify if the comparing segment column was used in the relevance analysis or not.
Main column from which the insight is created.
Selection Values.
In categorical and text columns, a selection of single values can be made and this type is useful to do that.
Each item in array.
Selection Range.
This type is useful for quantitative columns, in which a selection of values in range can be made.
Each item in array.
# create_correlations_insight
Source: https://docs.graphext.com/api-docs/report/create_insight/create_correlations_insight
Create a new insight from the Correlations section.
## Usage
The following shows how the step can be used in a recipe.
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
create_correlations_insight(, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Title of the insight.
Name of the column that will be compared with other columns specified in the `correlatedColumns` parameter. Placed in the x axis.
Name of the column that will be compared with other columns specified in the `correlatedColumns` parameter Placed in the y axis.
A list of columns the columnX or columnY will be compared with.
The insight will contain one chart per correlated column. If columnX and columnY are provided correlatedColumns needs to contain only columnY.
Each item in array.
Columns hidden from the Correlations section when the insight is replayed.
When replayed, the Correlation section will compare the target variable with all variables in
the dataset except these.
Each item in array.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Sets the correlation graph as expanded.
If true, the insight will contain an expanded chart in order to show the maximum number of categories possible.
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
Sets the visualization type.
Whether a heatmap or a bubble like chart.
Values must be one of the following:
* `heatmap`
* `bubble`
Sets Boxplot as visualization type.
If true, a combination of quantitative & categorical columns will use Boxplot as visualization type.
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
Element visibility.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript theme={null}
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript theme={null}
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript theme={null}
0 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript theme={null}
0 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
# create_filter_insight
Source: https://docs.graphext.com/api-docs/report/create_insight/create_filter_insight
Create a new insight from a selection of nodes.
## Usage
The following examples show how the step can be used in a recipe.
The following example is the most simple way to create a new insight. It just create an insight of the "age" variable by adding its chart and a title.
```stan theme={null}
create_filter_insight({
"column": "age",
"title": "Age distribution"
})
```
The following example creates an insight of the "age" variable by adding its chart, a title and making a filter selection from its 75 percentil to its maximum value.
```stan theme={null}
create_filter_insight({
"column": "age",
"title": "Most aged people",
"selection": "age: >= P75 AND <= MAX"
})
```
The following example creates an insight of the "cluster" variable by representing it as list.
```stan theme={null}
create_filter_insight({
"column": "cluster",
"title": "Cluster segments",
"columnViewModes": {
"cluster": "list"
}
})
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
create_filter_insight(, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Title of the insight.
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
Column used to color the UI.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Defines if the cross-filters mode is set to auto.
Defines if the cross-filters and significant variables should ignore null values.
Sets the visualization type to absolute or relative.
Useful to shown or not the statistics of the main variable/column.
A filter query.
See [Advanced query filters](https://docs.graphext.com/concepts/ds-concepts/filter-queries#advanced-filter-queries) for more information.
The order in which a variable's categories are displayed.
An object configuring for each column a method determining the order of its categories.
Order of the categories in a specific column.
Values must be one of the following:
* `BACKGROUND`
* `FOREGROUND`
* `UPLIFT`
* `TFIDF`
* `ORDINAL`
Configure categories to hide in the filter view of categorical variables.
Categories less frequent than the configured threshold will not appear in the UI.
A list of threshold configurations.
A categorical column can have two kinds of thresholds determining whether specific categories will be
hidden from its view in the UI: a minimum number of rows in the current *selection* below which a category
will be hidden, or a minimum number of rows in the *whole dataset* (*everything*).
The `thresholds` parameter should be a list containing 1 or 2 objects: the configuration of a *selection*
threshold, and/or the configuration of a threshold for *everything*.
array.
Configure categories to be discarded (hidden) in terms of their occurrence in the *whole dataset*.
Categories with a number (or percentage) of rows in the *whole dataset* less than `value` will be discarded (hidden from the variable's filter view).
Whether to apply the threshold to the current selection of rows or all rows in the dataset.
Whether to interpret the threshold value as an absolute (count) or percentage of rows.
Values must be one of the following:
* `ABSOLUTE`
* `PERCENTAGE`
Categories less frequent than this value will be discarded (hidden).
array.
Configure categories to be discarded (hidden) in terms of their occurrence in the *current selection*.
Categories with a number (or percentage) of rows in the current selection less than `value` will be discarded (hidden from the variable's filter view).
Whether to apply the threshold to the current selection of rows or all rows in the dataset.
Whether to interpret the threshold value as an absolute (count) or percentage of rows.
Values must be one of the following:
* `ABSOLUTE`
* `PERCENTAGE`
Categories less frequent than this value will be discarded (hidden).
array.
Configure categories to be discarded (hidden) in terms of their occurrence in the *whole dataset*.
Categories with a number (or percentage) of rows in the *whole dataset* less than `value` will be discarded (hidden from the variable's filter view).
Whether to apply the threshold to the current selection of rows or all rows in the dataset.
Whether to interpret the threshold value as an absolute (count) or percentage of rows.
Values must be one of the following:
* `ABSOLUTE`
* `PERCENTAGE`
Categories less frequent than this value will be discarded (hidden).
Configure categories to be discarded (hidden) in terms of their occurrence in the *current selection*.
Categories with a number (or percentage) of rows in the current selection less than `value` will be discarded (hidden from the variable's filter view).
Whether to apply the threshold to the current selection of rows or all rows in the dataset.
Whether to interpret the threshold value as an absolute (count) or percentage of rows.
Values must be one of the following:
* `ABSOLUTE`
* `PERCENTAGE`
Categories less frequent than this value will be discarded (hidden).
array.
Configure categories to be discarded (hidden) in terms of their occurrence in the *current selection*.
Categories with a number (or percentage) of rows in the current selection less than `value` will be discarded (hidden from the variable's filter view).
Whether to apply the threshold to the current selection of rows or all rows in the dataset.
Whether to interpret the threshold value as an absolute (count) or percentage of rows.
Values must be one of the following:
* `ABSOLUTE`
* `PERCENTAGE`
Categories less frequent than this value will be discarded (hidden).
Configure categories to be discarded (hidden) in terms of their occurrence in the *whole dataset*.
Categories with a number (or percentage) of rows in the *whole dataset* less than `value` will be discarded (hidden from the variable's filter view).
Whether to apply the threshold to the current selection of rows or all rows in the dataset.
Whether to interpret the threshold value as an absolute (count) or percentage of rows.
Values must be one of the following:
* `ABSOLUTE`
* `PERCENTAGE`
Categories less frequent than this value will be discarded (hidden).
Flag for internal usage identifying non-user configurations.
Main column from which the insight is created.
Visual representation mode of the insight's columns.
For each column select whether to show it as a list or bar chart.
By default, all column representations are "barChart".
One or more additional parameters.
Values must be one of the following:
* `barChart`
* `list`
* `{"country": "list", "vote intention": "list", "cluster": "barChart"}`
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
Element visibility.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript theme={null}
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript theme={null}
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript theme={null}
0 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript theme={null}
0 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
# create_graph_insight
Source: https://docs.graphext.com/api-docs/report/create_insight/create_graph_insight
Create a new insight from the Graph section.
## Usage
The following shows how the step can be used in a recipe.
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
create_graph_insight(, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Title of the insight.
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
Column used to color the UI.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Defines if the cross-filters mode is set to auto.
Defines if the cross-filters and significant variables should ignore null values.
Defines if the cross-filters mode is set to relative.
A filter query.
See [Advanced query filters](https://docs.graphext.com/concepts/ds-concepts/filter-queries#advanced-filter-queries) for more information.
The order in which a variable's categories are displayed.
An object configuring for each column a method determining the order of its categories.
Order of the categories in a specific column.
Values must be one of the following:
* `BACKGROUND`
* `FOREGROUND`
* `UPLIFT`
* `TFIDF`
* `ORDINAL`
Configure categories to hide in the filter view of categorical variables.
Categories less frequent than the configured threshold will not appear in the UI.
A list of threshold configurations.
A categorical column can have two kinds of thresholds determining whether specific categories will be
hidden from its view in the UI: a minimum number of rows in the current *selection* below which a category
will be hidden, or a minimum number of rows in the *whole dataset* (*everything*).
The `thresholds` parameter should be a list containing 1 or 2 objects: the configuration of a *selection*
threshold, and/or the configuration of a threshold for *everything*.
array.
Configure categories to be discarded (hidden) in terms of their occurrence in the *whole dataset*.
Categories with a number (or percentage) of rows in the *whole dataset* less than `value` will be discarded (hidden from the variable's filter view).
Whether to apply the threshold to the current selection of rows or all rows in the dataset.
Whether to interpret the threshold value as an absolute (count) or percentage of rows.
Values must be one of the following:
* `ABSOLUTE`
* `PERCENTAGE`
Categories less frequent than this value will be discarded (hidden).
array.
Configure categories to be discarded (hidden) in terms of their occurrence in the *current selection*.
Categories with a number (or percentage) of rows in the current selection less than `value` will be discarded (hidden from the variable's filter view).
Whether to apply the threshold to the current selection of rows or all rows in the dataset.
Whether to interpret the threshold value as an absolute (count) or percentage of rows.
Values must be one of the following:
* `ABSOLUTE`
* `PERCENTAGE`
Categories less frequent than this value will be discarded (hidden).
array.
Configure categories to be discarded (hidden) in terms of their occurrence in the *whole dataset*.
Categories with a number (or percentage) of rows in the *whole dataset* less than `value` will be discarded (hidden from the variable's filter view).
Whether to apply the threshold to the current selection of rows or all rows in the dataset.
Whether to interpret the threshold value as an absolute (count) or percentage of rows.
Values must be one of the following:
* `ABSOLUTE`
* `PERCENTAGE`
Categories less frequent than this value will be discarded (hidden).
Configure categories to be discarded (hidden) in terms of their occurrence in the *current selection*.
Categories with a number (or percentage) of rows in the current selection less than `value` will be discarded (hidden from the variable's filter view).
Whether to apply the threshold to the current selection of rows or all rows in the dataset.
Whether to interpret the threshold value as an absolute (count) or percentage of rows.
Values must be one of the following:
* `ABSOLUTE`
* `PERCENTAGE`
Categories less frequent than this value will be discarded (hidden).
array.
Configure categories to be discarded (hidden) in terms of their occurrence in the *current selection*.
Categories with a number (or percentage) of rows in the current selection less than `value` will be discarded (hidden from the variable's filter view).
Whether to apply the threshold to the current selection of rows or all rows in the dataset.
Whether to interpret the threshold value as an absolute (count) or percentage of rows.
Values must be one of the following:
* `ABSOLUTE`
* `PERCENTAGE`
Categories less frequent than this value will be discarded (hidden).
Configure categories to be discarded (hidden) in terms of their occurrence in the *whole dataset*.
Categories with a number (or percentage) of rows in the *whole dataset* less than `value` will be discarded (hidden from the variable's filter view).
Whether to apply the threshold to the current selection of rows or all rows in the dataset.
Whether to interpret the threshold value as an absolute (count) or percentage of rows.
Values must be one of the following:
* `ABSOLUTE`
* `PERCENTAGE`
Categories less frequent than this value will be discarded (hidden).
Flag for internal usage identifying non-user configurations.
Label of the graph chart.
Column used for the node sizes in the UI.
Visual representation mode of the insight's columns.
For each column select whether to show it as a list or bar chart.
By default, all column representations are "barChart".
One or more additional parameters.
Values must be one of the following:
* `barChart`
* `list`
* `{"country": "list", "vote intention": "list", "cluster": "barChart"}`
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
Element visibility.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript theme={null}
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript theme={null}
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript theme={null}
0 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript theme={null}
0 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
# create_plot_insight
Source: https://docs.graphext.com/api-docs/report/create_insight/create_plot_insight
Create a new insight from the Plot section.
## Usage
The following shows how the step can be used in a recipe.
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan
create_plot_insight(, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Area Chart.
Area Chart.
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Defines if the cross-filters mode is set to auto.
Defines if the cross-filters and significant variables should ignore null values.
Insight default layout.
Values must be one of the following:
* `plot`
* `all`
Insight default layout elements.
Each item in array.
Values must be one of the following:
* `title`
* `description`
* `filters`
Defines if the cross-filters mode is set to relative.
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
A filter query.
See [Advanced query filters](https://docs.graphext.com/concepts/ds-concepts/filter-queries#advanced-filter-queries) for more information.
Title of the insight.
Comments that can be added to the chart.
Any text can be added as annotation at certain abscissa point.
Alignment of the annotation.
Values must be one of the following:
* `start`
* `center`
* `end`
Text of the comment.
Annotations shape.
Annotations shape configuration.
Annotation shape type.
Values must be one of the following:
* `arrow`
Annotation endpoint type for arrow shape.
Values must be one of the following:
* `line`
* `solid`
* `none`
Annotation shape line style.
Values must be one of the following:
* `solid`
* `dashed`
Annotation shape line color.
Annotation shape line dash.
Each item in array.
Annotation shape line width.
Annotation shape endpoint position x for arrow shape.
Annotation shape endpoint position y for arrow shape.
Annotations style.
Annotation label style configuration.
Annotation label color.
Annotation label font size in pixels.
Annotation label font style.
Values must be one of the following:
* `bold`
* `italic`
* `normal`
X axis value.
X axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in X axis.
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in X axis.
Y axis value.
Y axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Values must be one of the following:
`max` `mean` `median` `min` `q1` `q3`
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in Y axis.
Chart description.
Chart footer text.
Chart size.
Chart width & height if customized, undefined if in "Fit to screen" mode.
string.
Values must be one of the following:
* `automatic`
Chart size width.
Chart size height.
Chart subtitle.
Chart title.
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Defines if the discretization in the X axis uses a exact or pretty number of bins.
Defines if the discretization in the X axis uses quantiles.
Chart markers foregroundColor.
Hides chart X axis title.
Hides chart Y axis title.
Interpolation method used for line or area chart.
Values must be one of the following:
`linear` `monotone` `cardinal` `natural` `step` `step-before` `step-after`
Stat configuration.
For example, for a max stat it would be:
`{ "stat": "max" }`
For stats that require a param like "countWhere" or "percentOfRowsWhere" it would be:
`{ "stat": "countWhere", "params": { "value": "categoryName" } }`
Stat kind.
Values must be one of the following:
`sum` `mean` `variance` `standardDeviation` `skewness` `kurtosis` `min` `p25` `p50` `p75` `max` `nNulls` `precision` `count` `cumSum` `nodeCount` `rForeground` `rForegroundColor` `rForegroundX` `uniqueValues` `valueCount` `countWhere` `percentOfRowsWhere`
Stat parameters.
Value for the stat parameter.
Theme applied to the chart.
Values must be one of the following:
`graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme color palette applied to the chart.
Values must be one of the following:
`Horus` `Osiris` `Re` `Blues` `Greens` `Oranges` `Purples` `Reds` `Viridis` `Magma` `Plasma` `Red-Blue` `Blue-Orange` `Red-Grey` `Red-Yellow-Blue` `Red-Yellow-Green` `graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme mode applied to the chart.
Values must be one of the following:
* `dark`
* `light`
Defines if tooltip is enabled.
Slot value represented in the x axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean, number or date, as well as listIndex.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
X Axis config options.
Configuration options specific to X axis, including base axis properties and label display controls.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis labels orientation mode.
Controls how axis labels are oriented. auto (rotates only when needed), rotate (always rotated), or none (never rotated).
Values must be one of the following:
* `auto`
* `rotate`
* `none`
Labels rotation in degrees.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart X axis title.
Slot value represented in the y axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean or number, as well as nodeCount or a metric.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Y Axis config options.
Configuration options specific to Y axis, including base axis properties.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart Y axis title.
Selection applied to chart in the x axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Selection applied to chart in the y axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Bar Chart.
Bar Chart.
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Defines if the cross-filters mode is set to auto.
Defines if the cross-filters and significant variables should ignore null values.
Insight default layout.
Values must be one of the following:
* `plot`
* `all`
Insight default layout elements.
Each item in array.
Values must be one of the following:
* `title`
* `description`
* `filters`
Defines if the cross-filters mode is set to relative.
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
A filter query.
See [Advanced query filters](https://docs.graphext.com/concepts/ds-concepts/filter-queries#advanced-filter-queries) for more information.
Title of the insight.
Comments that can be added to the chart.
Any text can be added as annotation at certain abscissa point.
Alignment of the annotation.
Values must be one of the following:
* `start`
* `center`
* `end`
Text of the comment.
Annotations shape.
Annotations shape configuration.
Annotation shape type.
Values must be one of the following:
* `arrow`
Annotation endpoint type for arrow shape.
Values must be one of the following:
* `line`
* `solid`
* `none`
Annotation shape line style.
Values must be one of the following:
* `solid`
* `dashed`
Annotation shape line color.
Annotation shape line dash.
Each item in array.
Annotation shape line width.
Annotation shape endpoint position x for arrow shape.
Annotation shape endpoint position y for arrow shape.
Annotations style.
Annotation label style configuration.
Annotation label color.
Annotation label font size in pixels.
Annotation label font style.
Values must be one of the following:
* `bold`
* `italic`
* `normal`
X axis value.
X axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in X axis.
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in X axis.
Y axis value.
Y axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Values must be one of the following:
`max` `mean` `median` `min` `q1` `q3`
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in Y axis.
Bars paddings in Bar charts.
Defines the inner & outer paddings for bars in Bar charts.
Inner padding between bars.
Outer padding for bars.
Chart description.
Defines the direction in which the chart is drawn.
Defines if Bar or BoxPlot charts will be drawn horizontally or vertically.
Values must be one of the following:
* `horizontal`
* `vertical`
Chart footer text.
Chart size.
Chart width & height if customized, undefined if in "Fit to screen" mode.
string.
Values must be one of the following:
* `automatic`
Chart size width.
Chart size height.
Chart subtitle.
Chart title.
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Defines if the discretization in the X axis uses a exact or pretty number of bins.
Defines if the discretization in the X axis uses quantiles.
Chart markers foregroundColor.
Hides chart X axis title.
Hides chart Y axis title.
Chart value labels.
Show & configure value labels in some charts.
Column for node labels in scatterplot charts.
Style configurations for value labels in bar charts.
Set position & alignment configuration for value labels in bar charts.
Value labels position in bar charts.
Values must be one of the following:
* `inside`
* `outside`
Value labels alignment in horizontal bar charts.
Values must be one of the following:
* `left`
* `middle`
* `right`
Format for value labels.
Format for absolute value labels.
Format for value count labels.
Format for percentage change labels.
Margin between node labels in scatterplot charts.
Whether to show value labels.
Whether to show absolute value labels.
Whether to show value count labels.
Whether to show percentage change labels.
Whether to show total value labels in stacked bar charts.
Metrics to show as value labels in boxplot charts.
Each item in array.
Chart sorting applied to chart in the x axis.
Values must be one of the following:
`XAXIS` `YAXIS` `YAXIS_SEGMENT` `SELECTION` `ORDINAL` `ALPHANUM`
Chart sorting direction applied to chart in the x axis.
Values must be one of the following:
* `ASC`
* `DESC`
Stat configuration.
For example, for a max stat it would be:
`{ "stat": "max" }`
For stats that require a param like "countWhere" or "percentOfRowsWhere" it would be:
`{ "stat": "countWhere", "params": { "value": "categoryName" } }`
Stat kind.
Values must be one of the following:
`sum` `mean` `variance` `standardDeviation` `skewness` `kurtosis` `min` `p25` `p50` `p75` `max` `nNulls` `precision` `count` `cumSum` `nodeCount` `rForeground` `rForegroundColor` `rForegroundX` `uniqueValues` `valueCount` `countWhere` `percentOfRowsWhere`
Stat parameters.
Value for the stat parameter.
Theme applied to the chart.
Values must be one of the following:
`graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme color palette applied to the chart.
Values must be one of the following:
`Horus` `Osiris` `Re` `Blues` `Greens` `Oranges` `Purples` `Reds` `Viridis` `Magma` `Plasma` `Red-Blue` `Blue-Orange` `Red-Grey` `Red-Yellow-Blue` `Red-Yellow-Green` `graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme mode applied to the chart.
Values must be one of the following:
* `dark`
* `light`
Defines if tooltip is enabled.
Slot value represented in the x axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean, number or date, as well as listIndex.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
X Axis config options.
Configuration options specific to X axis, including base axis properties and label display controls.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis labels orientation mode.
Controls how axis labels are oriented. auto (rotates only when needed), rotate (always rotated), or none (never rotated).
Values must be one of the following:
* `auto`
* `rotate`
* `none`
Labels rotation in degrees.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart X axis title.
Slot value represented in the y axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean or number, as well as nodeCount or a metric.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Y Axis config options.
Configuration options specific to Y axis, including base axis properties.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart Y axis title.
Selection applied to chart in the x axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Selection applied to chart in the y axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Box Plot Chart.
Box Plot Chart.
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Defines if the cross-filters mode is set to auto.
Defines if the cross-filters and significant variables should ignore null values.
Insight default layout.
Values must be one of the following:
* `plot`
* `all`
Insight default layout elements.
Each item in array.
Values must be one of the following:
* `title`
* `description`
* `filters`
Defines if the cross-filters mode is set to relative.
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
A filter query.
See [Advanced query filters](https://docs.graphext.com/concepts/ds-concepts/filter-queries#advanced-filter-queries) for more information.
Title of the insight.
Comments that can be added to the chart.
Any text can be added as annotation at certain abscissa point.
Alignment of the annotation.
Values must be one of the following:
* `start`
* `center`
* `end`
Text of the comment.
Annotations shape.
Annotations shape configuration.
Annotation shape type.
Values must be one of the following:
* `arrow`
Annotation endpoint type for arrow shape.
Values must be one of the following:
* `line`
* `solid`
* `none`
Annotation shape line style.
Values must be one of the following:
* `solid`
* `dashed`
Annotation shape line color.
Annotation shape line dash.
Each item in array.
Annotation shape line width.
Annotation shape endpoint position x for arrow shape.
Annotation shape endpoint position y for arrow shape.
Annotations style.
Annotation label style configuration.
Annotation label color.
Annotation label font size in pixels.
Annotation label font style.
Values must be one of the following:
* `bold`
* `italic`
* `normal`
X axis value.
X axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in X axis.
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in X axis.
Y axis value.
Y axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Values must be one of the following:
`max` `mean` `median` `min` `q1` `q3`
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in Y axis.
Defines the metrics to be drawn in BoxPlot charts.
Values must be one of the following:
* `quartiles`
* `quartiles+IQR`
* `quartiles+min/max`
* `mean+standardDeviation`
Chart description.
Defines the direction in which the chart is drawn.
Defines if Bar or BoxPlot charts will be drawn horizontally or vertically.
Values must be one of the following:
* `horizontal`
* `vertical`
Chart footer text.
Chart size.
Chart width & height if customized, undefined if in "Fit to screen" mode.
string.
Values must be one of the following:
* `automatic`
Chart size width.
Chart size height.
Chart subtitle.
Chart title.
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Defines if the discretization in the X axis uses a exact or pretty number of bins.
Defines if the discretization in the X axis uses quantiles.
Chart markers foregroundColor.
Hides chart X axis title.
Hides chart Y axis title.
Chart value labels.
Show & configure value labels in some charts.
Column for node labels in scatterplot charts.
Style configurations for value labels in bar charts.
Set position & alignment configuration for value labels in bar charts.
Value labels position in bar charts.
Values must be one of the following:
* `inside`
* `outside`
Value labels alignment in horizontal bar charts.
Values must be one of the following:
* `left`
* `middle`
* `right`
Format for value labels.
Format for absolute value labels.
Format for value count labels.
Format for percentage change labels.
Margin between node labels in scatterplot charts.
Whether to show value labels.
Whether to show absolute value labels.
Whether to show value count labels.
Whether to show percentage change labels.
Whether to show total value labels in stacked bar charts.
Metrics to show as value labels in boxplot charts.
Each item in array.
Chart sorting applied to chart in the x axis.
Values must be one of the following:
`XAXIS` `YAXIS` `YAXIS_SEGMENT` `SELECTION` `ORDINAL` `ALPHANUM`
Chart sorting direction applied to chart in the x axis.
Values must be one of the following:
* `ASC`
* `DESC`
Theme applied to the chart.
Values must be one of the following:
`graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme color palette applied to the chart.
Values must be one of the following:
`Horus` `Osiris` `Re` `Blues` `Greens` `Oranges` `Purples` `Reds` `Viridis` `Magma` `Plasma` `Red-Blue` `Blue-Orange` `Red-Grey` `Red-Yellow-Blue` `Red-Yellow-Green` `graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme mode applied to the chart.
Values must be one of the following:
* `dark`
* `light`
Defines if tooltip is enabled.
Slot value represented in the x axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean, number or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
X Axis config options.
Configuration options specific to X axis, including base axis properties and label display controls.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis labels orientation mode.
Controls how axis labels are oriented. auto (rotates only when needed), rotate (always rotated), or none (never rotated).
Values must be one of the following:
* `auto`
* `rotate`
* `none`
Labels rotation in degrees.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart X axis title.
Slot value represented in the y axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type number.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Y Axis config options.
Configuration options specific to Y axis, including base axis properties.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart Y axis title.
Selection applied to chart in the x axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Selection applied to chart in the y axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Bubble Chart.
Bubble Chart.
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Defines if the cross-filters mode is set to auto.
Defines if the cross-filters and significant variables should ignore null values.
Insight default layout.
Values must be one of the following:
* `plot`
* `all`
Insight default layout elements.
Each item in array.
Values must be one of the following:
* `title`
* `description`
* `filters`
Defines if the cross-filters mode is set to relative.
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
A filter query.
See [Advanced query filters](https://docs.graphext.com/concepts/ds-concepts/filter-queries#advanced-filter-queries) for more information.
Title of the insight.
Chart description.
Chart footer text.
Chart size.
Chart width & height if customized, undefined if in "Fit to screen" mode.
string.
Values must be one of the following:
* `automatic`
Chart size width.
Chart size height.
Chart subtitle.
Chart title.
Chart markers foregroundColor.
Hides chart size legend title.
Hides chart X axis title.
Hides chart Y axis title.
Chart value labels.
Show & configure value labels in some charts.
Column for node labels in scatterplot charts.
Style configurations for value labels in bar charts.
Set position & alignment configuration for value labels in bar charts.
Value labels position in bar charts.
Values must be one of the following:
* `inside`
* `outside`
Value labels alignment in horizontal bar charts.
Values must be one of the following:
* `left`
* `middle`
* `right`
Format for value labels.
Format for absolute value labels.
Format for value count labels.
Format for percentage change labels.
Margin between node labels in scatterplot charts.
Whether to show value labels.
Whether to show absolute value labels.
Whether to show value count labels.
Whether to show percentage change labels.
Whether to show total value labels in stacked bar charts.
Metrics to show as value labels in boxplot charts.
Each item in array.
Chart size legend title.
Nodes size in scatterplot charts.
Can define a fixed value or a min & max value in px to map nodes size.
number.
Minimum size value in px.
Maximum size value in px.
Nodes opacity in scatterplot charts.
Metrics shown in scatterplot charts.
Defines if regression line, RSquared or Pearson coefficient metrics are displayed.
Whether to show a regression line.
Whether to show RSquared metric.
Whether to show Pearson coefficient metric.
Slot value that maps bubble size in scatterplot charts.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type number or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Selection applied to the column that maps bubble size in scatterplot charts.
It can be an array with the categories selected in categorical columns, or an array with the range of the selection for quantitative or date columns.
array.
Each item in array.
array.
Each item in array.
Theme applied to the chart.
Values must be one of the following:
`graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme color palette applied to the chart.
Values must be one of the following:
`Horus` `Osiris` `Re` `Blues` `Greens` `Oranges` `Purples` `Reds` `Viridis` `Magma` `Plasma` `Red-Blue` `Blue-Orange` `Red-Grey` `Red-Yellow-Blue` `Red-Yellow-Green` `graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme mode applied to the chart.
Values must be one of the following:
* `dark`
* `light`
Defines if tooltip is enabled.
Fields in tooltip.
A list of fields to show in tooltip for Scatterplot charts.
Each item in array.
Slot value represented in the x axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type number or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
X Axis config options.
Configuration options specific to X axis, including base axis properties and label display controls.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis labels orientation mode.
Controls how axis labels are oriented. auto (rotates only when needed), rotate (always rotated), or none (never rotated).
Values must be one of the following:
* `auto`
* `rotate`
* `none`
Labels rotation in degrees.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart X axis title.
Slot value represented in the y axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type number or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Y Axis config options.
Configuration options specific to Y axis, including base axis properties.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart Y axis title.
Bubble Colored Chart.
Bubble Colored Chart.
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Defines if the cross-filters mode is set to auto.
Defines if the cross-filters and significant variables should ignore null values.
Insight default layout.
Values must be one of the following:
* `plot`
* `all`
Insight default layout elements.
Each item in array.
Values must be one of the following:
* `title`
* `description`
* `filters`
Defines if the cross-filters mode is set to relative.
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
A filter query.
See [Advanced query filters](https://docs.graphext.com/concepts/ds-concepts/filter-queries#advanced-filter-queries) for more information.
Title of the insight.
Chart description.
Chart footer text.
Chart size.
Chart width & height if customized, undefined if in "Fit to screen" mode.
string.
Values must be one of the following:
* `automatic`
Chart size width.
Chart size height.
Chart subtitle.
Chart title.
Slot value used to map color in the chart.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean, number or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Categories of the color variable.
Each item in array.
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Hides chart color legend title.
Hides chart size legend title.
Hides chart X axis title.
Hides chart Y axis title.
Chart value labels.
Show & configure value labels in some charts.
Column for node labels in scatterplot charts.
Style configurations for value labels in bar charts.
Set position & alignment configuration for value labels in bar charts.
Value labels position in bar charts.
Values must be one of the following:
* `inside`
* `outside`
Value labels alignment in horizontal bar charts.
Values must be one of the following:
* `left`
* `middle`
* `right`
Format for value labels.
Format for absolute value labels.
Format for value count labels.
Format for percentage change labels.
Margin between node labels in scatterplot charts.
Whether to show value labels.
Whether to show absolute value labels.
Whether to show value count labels.
Whether to show percentage change labels.
Whether to show total value labels in stacked bar charts.
Metrics to show as value labels in boxplot charts.
Each item in array.
Chart color legend title.
Chart size legend title.
Nodes size in scatterplot charts.
Can define a fixed value or a min & max value in px to map nodes size.
number.
Minimum size value in px.
Maximum size value in px.
Nodes opacity in scatterplot charts.
Metrics shown in scatterplot charts.
Defines if regression line, RSquared or Pearson coefficient metrics are displayed.
Whether to show a regression line.
Whether to show RSquared metric.
Whether to show Pearson coefficient metric.
Slot value that maps bubble size in scatterplot charts.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type number or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Selection applied to the column that maps bubble size in scatterplot charts.
It can be an array with the categories selected in categorical columns, or an array with the range of the selection for quantitative or date columns.
array.
Each item in array.
array.
Each item in array.
Sorting applied to categories of the color variable (segmentsColumn).
Values must be one of the following:
`XAXIS` `YAXIS` `YAXIS_SEGMENT` `SELECTION` `ORDINAL` `ALPHANUM`
Sorting direction applied to categories of the color variable (segmentsColumn).
Values must be one of the following:
* `ASC`
* `DESC`
Sorting mode applied to categories of the color variable (segmentsColumn). CUSTOM does not apply any sorting criteria.
Values must be one of the following:
* `SORT`
* `CUSTOM`
Theme applied to the chart.
Values must be one of the following:
`graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme color palette applied to the chart.
Values must be one of the following:
`Horus` `Osiris` `Re` `Blues` `Greens` `Oranges` `Purples` `Reds` `Viridis` `Magma` `Plasma` `Red-Blue` `Blue-Orange` `Red-Grey` `Red-Yellow-Blue` `Red-Yellow-Green` `graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme mode applied to the chart.
Values must be one of the following:
* `dark`
* `light`
Defines if tooltip is enabled.
Fields in tooltip.
A list of fields to show in tooltip for Scatterplot charts.
Each item in array.
Slot value represented in the x axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type number or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
X Axis config options.
Configuration options specific to X axis, including base axis properties and label display controls.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis labels orientation mode.
Controls how axis labels are oriented. auto (rotates only when needed), rotate (always rotated), or none (never rotated).
Values must be one of the following:
* `auto`
* `rotate`
* `none`
Labels rotation in degrees.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart X axis title.
Slot value represented in the y axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type number or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Y Axis config options.
Configuration options specific to Y axis, including base axis properties.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart Y axis title.
Grouped Bar Chart.
Grouped Bar Chart.
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Defines if the cross-filters mode is set to auto.
Defines if the cross-filters and significant variables should ignore null values.
Insight default layout.
Values must be one of the following:
* `plot`
* `all`
Insight default layout elements.
Each item in array.
Values must be one of the following:
* `title`
* `description`
* `filters`
Defines if the cross-filters mode is set to relative.
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
A filter query.
See [Advanced query filters](https://docs.graphext.com/concepts/ds-concepts/filter-queries#advanced-filter-queries) for more information.
Title of the insight.
Comments that can be added to the chart.
Any text can be added as annotation at certain abscissa point.
Alignment of the annotation.
Values must be one of the following:
* `start`
* `center`
* `end`
Text of the comment.
Annotations shape.
Annotations shape configuration.
Annotation shape type.
Values must be one of the following:
* `arrow`
Annotation endpoint type for arrow shape.
Values must be one of the following:
* `line`
* `solid`
* `none`
Annotation shape line style.
Values must be one of the following:
* `solid`
* `dashed`
Annotation shape line color.
Annotation shape line dash.
Each item in array.
Annotation shape line width.
Annotation shape endpoint position x for arrow shape.
Annotation shape endpoint position y for arrow shape.
Annotations style.
Annotation label style configuration.
Annotation label color.
Annotation label font size in pixels.
Annotation label font style.
Values must be one of the following:
* `bold`
* `italic`
* `normal`
X axis value.
X axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in X axis.
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in X axis.
Y axis value.
Y axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Values must be one of the following:
`max` `mean` `median` `min` `q1` `q3`
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in Y axis.
Bars paddings in Bar charts.
Defines the inner & outer paddings for bars in Bar charts.
Inner padding between bars.
Outer padding for bars.
Chart description.
Defines the direction in which the chart is drawn.
Defines if Bar or BoxPlot charts will be drawn horizontally or vertically.
Values must be one of the following:
* `horizontal`
* `vertical`
Chart footer text.
Chart size.
Chart width & height if customized, undefined if in "Fit to screen" mode.
string.
Values must be one of the following:
* `automatic`
Chart size width.
Chart size height.
Chart subtitle.
Chart title.
Slot value used to map color in the chart.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Categories of the color variable.
Each item in array.
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Defines if the discretization in the X axis uses a exact or pretty number of bins.
Defines if the discretization in the X axis uses quantiles.
Hides chart color legend title.
Hides chart X axis title.
Hides chart Y axis title.
Chart value labels.
Show & configure value labels in some charts.
Column for node labels in scatterplot charts.
Style configurations for value labels in bar charts.
Set position & alignment configuration for value labels in bar charts.
Value labels position in bar charts.
Values must be one of the following:
* `inside`
* `outside`
Value labels alignment in horizontal bar charts.
Values must be one of the following:
* `left`
* `middle`
* `right`
Format for value labels.
Format for absolute value labels.
Format for value count labels.
Format for percentage change labels.
Margin between node labels in scatterplot charts.
Whether to show value labels.
Whether to show absolute value labels.
Whether to show value count labels.
Whether to show percentage change labels.
Whether to show total value labels in stacked bar charts.
Metrics to show as value labels in boxplot charts.
Each item in array.
Chart color legend title.
Sorting applied to categories of the color variable (segmentsColumn).
Values must be one of the following:
`XAXIS` `YAXIS` `YAXIS_SEGMENT` `SELECTION` `ORDINAL` `ALPHANUM`
Sorting direction applied to categories of the color variable (segmentsColumn).
Values must be one of the following:
* `ASC`
* `DESC`
Sorting mode applied to categories of the color variable (segmentsColumn). CUSTOM does not apply any sorting criteria.
Values must be one of the following:
* `SORT`
* `CUSTOM`
Chart sorting applied to chart in the x axis.
Values must be one of the following:
`XAXIS` `YAXIS` `YAXIS_SEGMENT` `SELECTION` `ORDINAL` `ALPHANUM`
Chart sorting direction applied to chart in the x axis.
Values must be one of the following:
* `ASC`
* `DESC`
Stat configuration.
For example, for a max stat it would be:
`{ "stat": "max" }`
For stats that require a param like "countWhere" or "percentOfRowsWhere" it would be:
`{ "stat": "countWhere", "params": { "value": "categoryName" } }`
Stat kind.
Values must be one of the following:
`sum` `mean` `variance` `standardDeviation` `skewness` `kurtosis` `min` `p25` `p50` `p75` `max` `nNulls` `precision` `count` `cumSum` `nodeCount` `rForeground` `rForegroundColor` `rForegroundX` `uniqueValues` `valueCount` `countWhere` `percentOfRowsWhere`
Stat parameters.
Value for the stat parameter.
Theme applied to the chart.
Values must be one of the following:
`graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme color palette applied to the chart.
Values must be one of the following:
`Horus` `Osiris` `Re` `Blues` `Greens` `Oranges` `Purples` `Reds` `Viridis` `Magma` `Plasma` `Red-Blue` `Blue-Orange` `Red-Grey` `Red-Yellow-Blue` `Red-Yellow-Green` `graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme mode applied to the chart.
Values must be one of the following:
* `dark`
* `light`
Defines if tooltip is enabled.
Slot value represented in the x axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean, number or date, as well as listIndex.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
X Axis config options.
Configuration options specific to X axis, including base axis properties and label display controls.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis labels orientation mode.
Controls how axis labels are oriented. auto (rotates only when needed), rotate (always rotated), or none (never rotated).
Values must be one of the following:
* `auto`
* `rotate`
* `none`
Labels rotation in degrees.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart X axis title.
Slot value represented in the y axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean or number, as well as nodeCount or a metric.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Y Axis config options.
Configuration options specific to Y axis, including base axis properties.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart Y axis title.
Selection applied to chart in the x axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Selection applied to chart in the y axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Heat Map Chart.
Heat Map Chart.
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Defines if the cross-filters mode is set to auto.
Defines if the cross-filters and significant variables should ignore null values.
Insight default layout.
Values must be one of the following:
* `plot`
* `all`
Insight default layout elements.
Each item in array.
Values must be one of the following:
* `title`
* `description`
* `filters`
Defines if the cross-filters mode is set to relative.
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
A filter query.
See [Advanced query filters](https://docs.graphext.com/concepts/ds-concepts/filter-queries#advanced-filter-queries) for more information.
Title of the insight.
Slot value used to apply color to each cell in Heatmap charts.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean or number, as well as nodeCount or a metric.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Color legend title in Heatmap charts.
Chart description.
Chart footer text.
Chart size.
Chart width & height if customized, undefined if in "Fit to screen" mode.
string.
Values must be one of the following:
* `automatic`
Chart size width.
Chart size height.
Chart subtitle.
Chart title.
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Defines if the discretization in the X axis uses a exact or pretty number of bins.
Defines if the discretization in the Y axis uses a exact or pretty number of bins.
Defines if the discretization in the X axis uses quantiles.
Defines if the discretization in the Y axis uses quantiles.
Hides chart color legend title in Heatmap charts.
Hides chart X axis title.
Hides chart Y axis title.
Chart value labels.
Show & configure value labels in some charts.
Column for node labels in scatterplot charts.
Style configurations for value labels in bar charts.
Set position & alignment configuration for value labels in bar charts.
Value labels position in bar charts.
Values must be one of the following:
* `inside`
* `outside`
Value labels alignment in horizontal bar charts.
Values must be one of the following:
* `left`
* `middle`
* `right`
Format for value labels.
Format for absolute value labels.
Format for value count labels.
Format for percentage change labels.
Margin between node labels in scatterplot charts.
Whether to show value labels.
Whether to show absolute value labels.
Whether to show value count labels.
Whether to show percentage change labels.
Whether to show total value labels in stacked bar charts.
Metrics to show as value labels in boxplot charts.
Each item in array.
Chart sorting applied to chart in the x axis.
Values must be one of the following:
`XAXIS` `YAXIS` `YAXIS_SEGMENT` `SELECTION` `ORDINAL` `ALPHANUM`
Chart sorting direction applied to chart in the x axis.
Values must be one of the following:
* `ASC`
* `DESC`
Chart sorting applied to chart in the y axis.
Values must be one of the following:
`XAXIS` `YAXIS` `YAXIS_SEGMENT` `SELECTION` `ORDINAL` `ALPHANUM`
Chart sorting direction applied to chart in the y axis.
Values must be one of the following:
* `ASC`
* `DESC`
Stat configuration.
For example, for a max stat it would be:
`{ "stat": "max" }`
For stats that require a param like "countWhere" or "percentOfRowsWhere" it would be:
`{ "stat": "countWhere", "params": { "value": "categoryName" } }`
Stat kind.
Values must be one of the following:
`sum` `mean` `variance` `standardDeviation` `skewness` `kurtosis` `min` `p25` `p50` `p75` `max` `nNulls` `precision` `count` `cumSum` `nodeCount` `rForeground` `rForegroundColor` `rForegroundX` `uniqueValues` `valueCount` `countWhere` `percentOfRowsWhere`
Stat parameters.
Value for the stat parameter.
Theme applied to the chart.
Values must be one of the following:
`graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme color palette applied to the chart.
Values must be one of the following:
`Horus` `Osiris` `Re` `Blues` `Greens` `Oranges` `Purples` `Reds` `Viridis` `Magma` `Plasma` `Red-Blue` `Blue-Orange` `Red-Grey` `Red-Yellow-Blue` `Red-Yellow-Green` `graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme mode applied to the chart.
Values must be one of the following:
* `dark`
* `light`
Defines if tooltip is enabled.
Slot value represented in the x axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean, number or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
X Axis config options.
Configuration options specific to X axis, including base axis properties and label display controls.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis labels orientation mode.
Controls how axis labels are oriented. auto (rotates only when needed), rotate (always rotated), or none (never rotated).
Values must be one of the following:
* `auto`
* `rotate`
* `none`
Labels rotation in degrees.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart X axis title.
Slot value represented in the y axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean, number or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Y Axis config options.
Configuration options specific to Y axis, including base axis properties.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart Y axis title.
Selection applied to chart in the x axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Selection applied to chart in the y axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Line Chart.
Line Chart.
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Defines if the cross-filters mode is set to auto.
Defines if the cross-filters and significant variables should ignore null values.
Insight default layout.
Values must be one of the following:
* `plot`
* `all`
Insight default layout elements.
Each item in array.
Values must be one of the following:
* `title`
* `description`
* `filters`
Defines if the cross-filters mode is set to relative.
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
A filter query.
See [Advanced query filters](https://docs.graphext.com/concepts/ds-concepts/filter-queries#advanced-filter-queries) for more information.
Title of the insight.
Comments that can be added to the chart.
Any text can be added as annotation at certain abscissa point.
Alignment of the annotation.
Values must be one of the following:
* `start`
* `center`
* `end`
Text of the comment.
Annotations shape.
Annotations shape configuration.
Annotation shape type.
Values must be one of the following:
* `arrow`
Annotation endpoint type for arrow shape.
Values must be one of the following:
* `line`
* `solid`
* `none`
Annotation shape line style.
Values must be one of the following:
* `solid`
* `dashed`
Annotation shape line color.
Annotation shape line dash.
Each item in array.
Annotation shape line width.
Annotation shape endpoint position x for arrow shape.
Annotation shape endpoint position y for arrow shape.
Annotations style.
Annotation label style configuration.
Annotation label color.
Annotation label font size in pixels.
Annotation label font style.
Values must be one of the following:
* `bold`
* `italic`
* `normal`
X axis value.
X axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in X axis.
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in X axis.
Y axis value.
Y axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Values must be one of the following:
`max` `mean` `median` `min` `q1` `q3`
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in Y axis.
Chart description.
Chart footer text.
Chart size.
Chart width & height if customized, undefined if in "Fit to screen" mode.
string.
Values must be one of the following:
* `automatic`
Chart size width.
Chart size height.
Chart subtitle.
Chart title.
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Defines if the discretization in the X axis uses a exact or pretty number of bins.
Defines if the discretization in the X axis uses quantiles.
Chart markers foregroundColor.
Hides chart X axis title.
Hides chart Y axis title.
Interpolation method used for line or area chart.
Values must be one of the following:
`linear` `monotone` `cardinal` `natural` `step` `step-before` `step-after`
Chart value labels.
Show & configure value labels in some charts.
Column for node labels in scatterplot charts.
Style configurations for value labels in bar charts.
Set position & alignment configuration for value labels in bar charts.
Value labels position in bar charts.
Values must be one of the following:
* `inside`
* `outside`
Value labels alignment in horizontal bar charts.
Values must be one of the following:
* `left`
* `middle`
* `right`
Format for value labels.
Format for absolute value labels.
Format for value count labels.
Format for percentage change labels.
Margin between node labels in scatterplot charts.
Whether to show value labels.
Whether to show absolute value labels.
Whether to show value count labels.
Whether to show percentage change labels.
Whether to show total value labels in stacked bar charts.
Metrics to show as value labels in boxplot charts.
Each item in array.
Line dash pattern.
Configures the dash pattern for lines in line charts.
Default dash pattern for lines.
It is represented as an array of stroke and gap lengths.
null.
array.
Each item in array.
Custom dash pattern.
Allows setting a specific dash pattern for the selected categories.
The custom dash pattern for the selected category.
It is represented as an array of stroke and gap lengths.
null.
array.
Each item in array.
Line markers.
Configures the markers for lines in line charts.
Line marker.
Configures a line marker.
The shape of the marker.
Values must be one of the following:
* `circle`
* `square`
* `triangle`
The color of the marker.
The area of the marker in pixels.
Custom line markers.
Allows setting a specific line marker for the selected categories.
Line marker.
Configures a line marker.
The shape of the marker.
Values must be one of the following:
* `circle`
* `square`
* `triangle`
The color of the marker.
The area of the marker in pixels.
Line width settings.
Specifies the width of the lines in line charts.
Default line width.
Custom line width.
Allows setting a specific line width for the selected categories.
The custom line width for the selected category.
Defines how to represent null values in line charts.
Values must be one of the following:
* `interpolated`
* `gap`
* `zero`
Stat configuration.
For example, for a max stat it would be:
`{ "stat": "max" }`
For stats that require a param like "countWhere" or "percentOfRowsWhere" it would be:
`{ "stat": "countWhere", "params": { "value": "categoryName" } }`
Stat kind.
Values must be one of the following:
`sum` `mean` `variance` `standardDeviation` `skewness` `kurtosis` `min` `p25` `p50` `p75` `max` `nNulls` `precision` `count` `cumSum` `nodeCount` `rForeground` `rForegroundColor` `rForegroundX` `uniqueValues` `valueCount` `countWhere` `percentOfRowsWhere`
Stat parameters.
Value for the stat parameter.
Theme applied to the chart.
Values must be one of the following:
`graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme color palette applied to the chart.
Values must be one of the following:
`Horus` `Osiris` `Re` `Blues` `Greens` `Oranges` `Purples` `Reds` `Viridis` `Magma` `Plasma` `Red-Blue` `Blue-Orange` `Red-Grey` `Red-Yellow-Blue` `Red-Yellow-Green` `graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme mode applied to the chart.
Values must be one of the following:
* `dark`
* `light`
Defines if tooltip is enabled.
Slot value represented in the x axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean, number or date, as well as listIndex.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
X Axis config options.
Configuration options specific to X axis, including base axis properties and label display controls.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis labels orientation mode.
Controls how axis labels are oriented. auto (rotates only when needed), rotate (always rotated), or none (never rotated).
Values must be one of the following:
* `auto`
* `rotate`
* `none`
Labels rotation in degrees.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart X axis title.
Slot value represented in the y axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean or number, as well as nodeCount or a metric.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Y Axis config options.
Configuration options specific to Y axis, including base axis properties.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart Y axis title.
Defines if zero values should be represented as null in line charts.
Selection applied to chart in the x axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Selection applied to chart in the y axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Multiple Line Chart.
Multiple Line Chart.
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Defines if the cross-filters mode is set to auto.
Defines if the cross-filters and significant variables should ignore null values.
Insight default layout.
Values must be one of the following:
* `plot`
* `all`
Insight default layout elements.
Each item in array.
Values must be one of the following:
* `title`
* `description`
* `filters`
Defines if the cross-filters mode is set to relative.
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
A filter query.
See [Advanced query filters](https://docs.graphext.com/concepts/ds-concepts/filter-queries#advanced-filter-queries) for more information.
Title of the insight.
Comments that can be added to the chart.
Any text can be added as annotation at certain abscissa point.
Alignment of the annotation.
Values must be one of the following:
* `start`
* `center`
* `end`
Text of the comment.
Annotations shape.
Annotations shape configuration.
Annotation shape type.
Values must be one of the following:
* `arrow`
Annotation endpoint type for arrow shape.
Values must be one of the following:
* `line`
* `solid`
* `none`
Annotation shape line style.
Values must be one of the following:
* `solid`
* `dashed`
Annotation shape line color.
Annotation shape line dash.
Each item in array.
Annotation shape line width.
Annotation shape endpoint position x for arrow shape.
Annotation shape endpoint position y for arrow shape.
Annotations style.
Annotation label style configuration.
Annotation label color.
Annotation label font size in pixels.
Annotation label font style.
Values must be one of the following:
* `bold`
* `italic`
* `normal`
X axis value.
X axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in X axis.
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in X axis.
Y axis value.
Y axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Values must be one of the following:
`max` `mean` `median` `min` `q1` `q3`
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in Y axis.
Chart description.
Chart footer text.
Chart size.
Chart width & height if customized, undefined if in "Fit to screen" mode.
string.
Values must be one of the following:
* `automatic`
Chart size width.
Chart size height.
Chart subtitle.
Chart title.
Slot value used to map color in the chart.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Categories of the color variable.
Each item in array.
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Defines if the discretization in the X axis uses a exact or pretty number of bins.
Defines if the discretization in the X axis uses quantiles.
Hides chart color legend title.
Hides chart X axis title.
Hides chart Y axis title.
Interpolation method used for line or area chart.
Values must be one of the following:
`linear` `monotone` `cardinal` `natural` `step` `step-before` `step-after`
Chart value labels.
Show & configure value labels in some charts.
Column for node labels in scatterplot charts.
Style configurations for value labels in bar charts.
Set position & alignment configuration for value labels in bar charts.
Value labels position in bar charts.
Values must be one of the following:
* `inside`
* `outside`
Value labels alignment in horizontal bar charts.
Values must be one of the following:
* `left`
* `middle`
* `right`
Format for value labels.
Format for absolute value labels.
Format for value count labels.
Format for percentage change labels.
Margin between node labels in scatterplot charts.
Whether to show value labels.
Whether to show absolute value labels.
Whether to show value count labels.
Whether to show percentage change labels.
Whether to show total value labels in stacked bar charts.
Metrics to show as value labels in boxplot charts.
Each item in array.
Chart color legend title.
Line dash pattern.
Configures the dash pattern for lines in line charts.
Default dash pattern for lines.
It is represented as an array of stroke and gap lengths.
null.
array.
Each item in array.
Custom dash pattern.
Allows setting a specific dash pattern for the selected categories.
The custom dash pattern for the selected category.
It is represented as an array of stroke and gap lengths.
null.
array.
Each item in array.
Line markers.
Configures the markers for lines in line charts.
Line marker.
Configures a line marker.
The shape of the marker.
Values must be one of the following:
* `circle`
* `square`
* `triangle`
The color of the marker.
The area of the marker in pixels.
Custom line markers.
Allows setting a specific line marker for the selected categories.
Line marker.
Configures a line marker.
The shape of the marker.
Values must be one of the following:
* `circle`
* `square`
* `triangle`
The color of the marker.
The area of the marker in pixels.
Line width settings.
Specifies the width of the lines in line charts.
Default line width.
Custom line width.
Allows setting a specific line width for the selected categories.
The custom line width for the selected category.
Defines how to represent null values in line charts.
Values must be one of the following:
* `interpolated`
* `gap`
* `zero`
Sorting applied to categories of the color variable (segmentsColumn).
Values must be one of the following:
`XAXIS` `YAXIS` `YAXIS_SEGMENT` `SELECTION` `ORDINAL` `ALPHANUM`
Sorting direction applied to categories of the color variable (segmentsColumn).
Values must be one of the following:
* `ASC`
* `DESC`
Sorting mode applied to categories of the color variable (segmentsColumn). CUSTOM does not apply any sorting criteria.
Values must be one of the following:
* `SORT`
* `CUSTOM`
Stat configuration.
For example, for a max stat it would be:
`{ "stat": "max" }`
For stats that require a param like "countWhere" or "percentOfRowsWhere" it would be:
`{ "stat": "countWhere", "params": { "value": "categoryName" } }`
Stat kind.
Values must be one of the following:
`sum` `mean` `variance` `standardDeviation` `skewness` `kurtosis` `min` `p25` `p50` `p75` `max` `nNulls` `precision` `count` `cumSum` `nodeCount` `rForeground` `rForegroundColor` `rForegroundX` `uniqueValues` `valueCount` `countWhere` `percentOfRowsWhere`
Stat parameters.
Value for the stat parameter.
Theme applied to the chart.
Values must be one of the following:
`graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme color palette applied to the chart.
Values must be one of the following:
`Horus` `Osiris` `Re` `Blues` `Greens` `Oranges` `Purples` `Reds` `Viridis` `Magma` `Plasma` `Red-Blue` `Blue-Orange` `Red-Grey` `Red-Yellow-Blue` `Red-Yellow-Green` `graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme mode applied to the chart.
Values must be one of the following:
* `dark`
* `light`
Defines if tooltip is enabled.
Slot value represented in the x axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean, number or date, as well as listIndex.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
X Axis config options.
Configuration options specific to X axis, including base axis properties and label display controls.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis labels orientation mode.
Controls how axis labels are oriented. auto (rotates only when needed), rotate (always rotated), or none (never rotated).
Values must be one of the following:
* `auto`
* `rotate`
* `none`
Labels rotation in degrees.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart X axis title.
Slot value represented in the y axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean or number, as well as nodeCount or a metric.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Y Axis config options.
Configuration options specific to Y axis, including base axis properties.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart Y axis title.
Defines if zero values should be represented as null in line charts.
Selection applied to chart in the x axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Selection applied to chart in the y axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Scatterplot Chart.
Scatterplot Chart.
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Defines if the cross-filters mode is set to auto.
Defines if the cross-filters and significant variables should ignore null values.
Insight default layout.
Values must be one of the following:
* `plot`
* `all`
Insight default layout elements.
Each item in array.
Values must be one of the following:
* `title`
* `description`
* `filters`
Defines if the cross-filters mode is set to relative.
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
A filter query.
See [Advanced query filters](https://docs.graphext.com/concepts/ds-concepts/filter-queries#advanced-filter-queries) for more information.
Title of the insight.
Chart description.
Chart footer text.
Chart size.
Chart width & height if customized, undefined if in "Fit to screen" mode.
string.
Values must be one of the following:
* `automatic`
Chart size width.
Chart size height.
Chart subtitle.
Chart title.
Chart markers foregroundColor.
Hides chart X axis title.
Hides chart Y axis title.
Chart value labels.
Show & configure value labels in some charts.
Column for node labels in scatterplot charts.
Style configurations for value labels in bar charts.
Set position & alignment configuration for value labels in bar charts.
Value labels position in bar charts.
Values must be one of the following:
* `inside`
* `outside`
Value labels alignment in horizontal bar charts.
Values must be one of the following:
* `left`
* `middle`
* `right`
Format for value labels.
Format for absolute value labels.
Format for value count labels.
Format for percentage change labels.
Margin between node labels in scatterplot charts.
Whether to show value labels.
Whether to show absolute value labels.
Whether to show value count labels.
Whether to show percentage change labels.
Whether to show total value labels in stacked bar charts.
Metrics to show as value labels in boxplot charts.
Each item in array.
Nodes size in scatterplot charts.
Can define a fixed value or a min & max value in px to map nodes size.
number.
Minimum size value in px.
Maximum size value in px.
Nodes opacity in scatterplot charts.
Metrics shown in scatterplot charts.
Defines if regression line, RSquared or Pearson coefficient metrics are displayed.
Whether to show a regression line.
Whether to show RSquared metric.
Whether to show Pearson coefficient metric.
Theme applied to the chart.
Values must be one of the following:
`graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme color palette applied to the chart.
Values must be one of the following:
`Horus` `Osiris` `Re` `Blues` `Greens` `Oranges` `Purples` `Reds` `Viridis` `Magma` `Plasma` `Red-Blue` `Blue-Orange` `Red-Grey` `Red-Yellow-Blue` `Red-Yellow-Green` `graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme mode applied to the chart.
Values must be one of the following:
* `dark`
* `light`
Defines if tooltip is enabled.
Fields in tooltip.
A list of fields to show in tooltip for Scatterplot charts.
Each item in array.
Slot value represented in the x axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type number or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
X Axis config options.
Configuration options specific to X axis, including base axis properties and label display controls.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis labels orientation mode.
Controls how axis labels are oriented. auto (rotates only when needed), rotate (always rotated), or none (never rotated).
Values must be one of the following:
* `auto`
* `rotate`
* `none`
Labels rotation in degrees.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart X axis title.
Slot value represented in the y axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type number or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Y Axis config options.
Configuration options specific to Y axis, including base axis properties.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart Y axis title.
Scatterplot Colored Chart.
Scatterplot Colored Chart.
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Defines if the cross-filters mode is set to auto.
Defines if the cross-filters and significant variables should ignore null values.
Insight default layout.
Values must be one of the following:
* `plot`
* `all`
Insight default layout elements.
Each item in array.
Values must be one of the following:
* `title`
* `description`
* `filters`
Defines if the cross-filters mode is set to relative.
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
A filter query.
See [Advanced query filters](https://docs.graphext.com/concepts/ds-concepts/filter-queries#advanced-filter-queries) for more information.
Title of the insight.
Chart description.
Chart footer text.
Chart size.
Chart width & height if customized, undefined if in "Fit to screen" mode.
string.
Values must be one of the following:
* `automatic`
Chart size width.
Chart size height.
Chart subtitle.
Chart title.
Slot value used to map color in the chart.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean, number or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Categories of the color variable.
Each item in array.
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Hides chart color legend title.
Hides chart X axis title.
Hides chart Y axis title.
Chart value labels.
Show & configure value labels in some charts.
Column for node labels in scatterplot charts.
Style configurations for value labels in bar charts.
Set position & alignment configuration for value labels in bar charts.
Value labels position in bar charts.
Values must be one of the following:
* `inside`
* `outside`
Value labels alignment in horizontal bar charts.
Values must be one of the following:
* `left`
* `middle`
* `right`
Format for value labels.
Format for absolute value labels.
Format for value count labels.
Format for percentage change labels.
Margin between node labels in scatterplot charts.
Whether to show value labels.
Whether to show absolute value labels.
Whether to show value count labels.
Whether to show percentage change labels.
Whether to show total value labels in stacked bar charts.
Metrics to show as value labels in boxplot charts.
Each item in array.
Chart color legend title.
Nodes size in scatterplot charts.
Can define a fixed value or a min & max value in px to map nodes size.
number.
Minimum size value in px.
Maximum size value in px.
Nodes opacity in scatterplot charts.
Metrics shown in scatterplot charts.
Defines if regression line, RSquared or Pearson coefficient metrics are displayed.
Whether to show a regression line.
Whether to show RSquared metric.
Whether to show Pearson coefficient metric.
Sorting applied to categories of the color variable (segmentsColumn).
Values must be one of the following:
`XAXIS` `YAXIS` `YAXIS_SEGMENT` `SELECTION` `ORDINAL` `ALPHANUM`
Sorting direction applied to categories of the color variable (segmentsColumn).
Values must be one of the following:
* `ASC`
* `DESC`
Sorting mode applied to categories of the color variable (segmentsColumn). CUSTOM does not apply any sorting criteria.
Values must be one of the following:
* `SORT`
* `CUSTOM`
Theme applied to the chart.
Values must be one of the following:
`graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme color palette applied to the chart.
Values must be one of the following:
`Horus` `Osiris` `Re` `Blues` `Greens` `Oranges` `Purples` `Reds` `Viridis` `Magma` `Plasma` `Red-Blue` `Blue-Orange` `Red-Grey` `Red-Yellow-Blue` `Red-Yellow-Green` `graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme mode applied to the chart.
Values must be one of the following:
* `dark`
* `light`
Defines if tooltip is enabled.
Fields in tooltip.
A list of fields to show in tooltip for Scatterplot charts.
Each item in array.
Slot value represented in the x axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type number or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
X Axis config options.
Configuration options specific to X axis, including base axis properties and label display controls.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis labels orientation mode.
Controls how axis labels are oriented. auto (rotates only when needed), rotate (always rotated), or none (never rotated).
Values must be one of the following:
* `auto`
* `rotate`
* `none`
Labels rotation in degrees.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart X axis title.
Slot value represented in the y axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type number or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Y Axis config options.
Configuration options specific to Y axis, including base axis properties.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart Y axis title.
Seasonal Decomposition Chart.
Seasonal Decomposition Chart.
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Defines if the cross-filters mode is set to auto.
Defines if the cross-filters and significant variables should ignore null values.
Insight default layout.
Values must be one of the following:
* `plot`
* `all`
Insight default layout elements.
Each item in array.
Values must be one of the following:
* `title`
* `description`
* `filters`
Defines if the cross-filters mode is set to relative.
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
A filter query.
See [Advanced query filters](https://docs.graphext.com/concepts/ds-concepts/filter-queries#advanced-filter-queries) for more information.
Title of the insight.
Comments that can be added to the chart.
Any text can be added as annotation at certain abscissa point.
Alignment of the annotation.
Values must be one of the following:
* `start`
* `center`
* `end`
Text of the comment.
Annotations shape.
Annotations shape configuration.
Annotation shape type.
Values must be one of the following:
* `arrow`
Annotation endpoint type for arrow shape.
Values must be one of the following:
* `line`
* `solid`
* `none`
Annotation shape line style.
Values must be one of the following:
* `solid`
* `dashed`
Annotation shape line color.
Annotation shape line dash.
Each item in array.
Annotation shape line width.
Annotation shape endpoint position x for arrow shape.
Annotation shape endpoint position y for arrow shape.
Annotations style.
Annotation label style configuration.
Annotation label color.
Annotation label font size in pixels.
Annotation label font style.
Values must be one of the following:
* `bold`
* `italic`
* `normal`
X axis value.
X axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in X axis.
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in X axis.
Y axis value.
Y axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Values must be one of the following:
`max` `mean` `median` `min` `q1` `q3`
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in Y axis.
Chart description.
Chart footer text.
Chart size.
Chart width & height if customized, undefined if in "Fit to screen" mode.
string.
Values must be one of the following:
* `automatic`
Chart size width.
Chart size height.
Chart subtitle.
Chart title.
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Defines if the discretization in the X axis uses a exact or pretty number of bins.
Defines if the discretization in the X axis uses quantiles.
Hides chart X axis title.
Hides chart Y axis title.
Interpolation method used for line or area chart.
Values must be one of the following:
`linear` `monotone` `cardinal` `natural` `step` `step-before` `step-after`
Seasonality interval for Seasonal Decomposition plot.
expanded segment in Seasonal Decomposition plot.
Stat configuration.
For example, for a max stat it would be:
`{ "stat": "max" }`
For stats that require a param like "countWhere" or "percentOfRowsWhere" it would be:
`{ "stat": "countWhere", "params": { "value": "categoryName" } }`
Stat kind.
Values must be one of the following:
`sum` `mean` `variance` `standardDeviation` `skewness` `kurtosis` `min` `p25` `p50` `p75` `max` `nNulls` `precision` `count` `cumSum` `nodeCount` `rForeground` `rForegroundColor` `rForegroundX` `uniqueValues` `valueCount` `countWhere` `percentOfRowsWhere`
Stat parameters.
Value for the stat parameter.
Theme applied to the chart.
Values must be one of the following:
`graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme color palette applied to the chart.
Values must be one of the following:
`Horus` `Osiris` `Re` `Blues` `Greens` `Oranges` `Purples` `Reds` `Viridis` `Magma` `Plasma` `Red-Blue` `Blue-Orange` `Red-Grey` `Red-Yellow-Blue` `Red-Yellow-Green` `graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme mode applied to the chart.
Values must be one of the following:
* `dark`
* `light`
Defines if tooltip is enabled.
Slot value represented in the x axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
X Axis config options.
Configuration options specific to X axis, including base axis properties and label display controls.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis labels orientation mode.
Controls how axis labels are oriented. auto (rotates only when needed), rotate (always rotated), or none (never rotated).
Values must be one of the following:
* `auto`
* `rotate`
* `none`
Labels rotation in degrees.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart X axis title.
Slot value represented in the y axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean or number, as well as nodeCount.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Y Axis config options.
Configuration options specific to Y axis, including base axis properties.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart Y axis title.
Defines if zero values should be represented as null in line charts.
Selection applied to chart in the x axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Segmented Area Chart.
Segmented Area Chart.
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Defines if the cross-filters mode is set to auto.
Defines if the cross-filters and significant variables should ignore null values.
Insight default layout.
Values must be one of the following:
* `plot`
* `all`
Insight default layout elements.
Each item in array.
Values must be one of the following:
* `title`
* `description`
* `filters`
Defines if the cross-filters mode is set to relative.
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
A filter query.
See [Advanced query filters](https://docs.graphext.com/concepts/ds-concepts/filter-queries#advanced-filter-queries) for more information.
Title of the insight.
Comments that can be added to the chart.
Any text can be added as annotation at certain abscissa point.
Alignment of the annotation.
Values must be one of the following:
* `start`
* `center`
* `end`
Text of the comment.
Annotations shape.
Annotations shape configuration.
Annotation shape type.
Values must be one of the following:
* `arrow`
Annotation endpoint type for arrow shape.
Values must be one of the following:
* `line`
* `solid`
* `none`
Annotation shape line style.
Values must be one of the following:
* `solid`
* `dashed`
Annotation shape line color.
Annotation shape line dash.
Each item in array.
Annotation shape line width.
Annotation shape endpoint position x for arrow shape.
Annotation shape endpoint position y for arrow shape.
Annotations style.
Annotation label style configuration.
Annotation label color.
Annotation label font size in pixels.
Annotation label font style.
Values must be one of the following:
* `bold`
* `italic`
* `normal`
X axis value.
X axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in X axis.
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in X axis.
Y axis value.
Y axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Values must be one of the following:
`max` `mean` `median` `min` `q1` `q3`
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in Y axis.
Chart description.
Chart footer text.
Chart size.
Chart width & height if customized, undefined if in "Fit to screen" mode.
string.
Values must be one of the following:
* `automatic`
Chart size width.
Chart size height.
Chart subtitle.
Chart title.
Slot value used to map color in the chart.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Categories of the color variable.
Each item in array.
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Defines if the discretization in the X axis uses a exact or pretty number of bins.
Defines if the discretization in the X axis uses quantiles.
Hides chart color legend title.
Hides chart X axis title.
Hides chart Y axis title.
Interpolation method used for line or area chart.
Values must be one of the following:
`linear` `monotone` `cardinal` `natural` `step` `step-before` `step-after`
Chart color legend title.
Sorting applied to categories of the color variable (segmentsColumn).
Values must be one of the following:
`XAXIS` `YAXIS` `YAXIS_SEGMENT` `SELECTION` `ORDINAL` `ALPHANUM`
Sorting direction applied to categories of the color variable (segmentsColumn).
Values must be one of the following:
* `ASC`
* `DESC`
Sorting mode applied to categories of the color variable (segmentsColumn). CUSTOM does not apply any sorting criteria.
Values must be one of the following:
* `SORT`
* `CUSTOM`
Stat configuration.
For example, for a max stat it would be:
`{ "stat": "max" }`
For stats that require a param like "countWhere" or "percentOfRowsWhere" it would be:
`{ "stat": "countWhere", "params": { "value": "categoryName" } }`
Stat kind.
Values must be one of the following:
`sum` `mean` `variance` `standardDeviation` `skewness` `kurtosis` `min` `p25` `p50` `p75` `max` `nNulls` `precision` `count` `cumSum` `nodeCount` `rForeground` `rForegroundColor` `rForegroundX` `uniqueValues` `valueCount` `countWhere` `percentOfRowsWhere`
Stat parameters.
Value for the stat parameter.
Theme applied to the chart.
Values must be one of the following:
`graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme color palette applied to the chart.
Values must be one of the following:
`Horus` `Osiris` `Re` `Blues` `Greens` `Oranges` `Purples` `Reds` `Viridis` `Magma` `Plasma` `Red-Blue` `Blue-Orange` `Red-Grey` `Red-Yellow-Blue` `Red-Yellow-Green` `graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme mode applied to the chart.
Values must be one of the following:
* `dark`
* `light`
Defines if tooltip is enabled.
Slot value represented in the x axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean, number or date, as well as listIndex.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
X Axis config options.
Configuration options specific to X axis, including base axis properties and label display controls.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis labels orientation mode.
Controls how axis labels are oriented. auto (rotates only when needed), rotate (always rotated), or none (never rotated).
Values must be one of the following:
* `auto`
* `rotate`
* `none`
Labels rotation in degrees.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart X axis title.
Slot value represented in the y axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean or number, as well as nodeCount or a metric.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Y Axis config options.
Configuration options specific to Y axis, including base axis properties.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart Y axis title.
Selection applied to chart in the x axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Selection applied to chart in the y axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Segmented Bar Chart.
Segmented Bar Chart.
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Defines if the cross-filters mode is set to auto.
Defines if the cross-filters and significant variables should ignore null values.
Insight default layout.
Values must be one of the following:
* `plot`
* `all`
Insight default layout elements.
Each item in array.
Values must be one of the following:
* `title`
* `description`
* `filters`
Defines if the cross-filters mode is set to relative.
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
A filter query.
See [Advanced query filters](https://docs.graphext.com/concepts/ds-concepts/filter-queries#advanced-filter-queries) for more information.
Title of the insight.
Comments that can be added to the chart.
Any text can be added as annotation at certain abscissa point.
Alignment of the annotation.
Values must be one of the following:
* `start`
* `center`
* `end`
Text of the comment.
Annotations shape.
Annotations shape configuration.
Annotation shape type.
Values must be one of the following:
* `arrow`
Annotation endpoint type for arrow shape.
Values must be one of the following:
* `line`
* `solid`
* `none`
Annotation shape line style.
Values must be one of the following:
* `solid`
* `dashed`
Annotation shape line color.
Annotation shape line dash.
Each item in array.
Annotation shape line width.
Annotation shape endpoint position x for arrow shape.
Annotation shape endpoint position y for arrow shape.
Annotations style.
Annotation label style configuration.
Annotation label color.
Annotation label font size in pixels.
Annotation label font style.
Values must be one of the following:
* `bold`
* `italic`
* `normal`
X axis value.
X axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in X axis.
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in X axis.
Y axis value.
Y axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Values must be one of the following:
`max` `mean` `median` `min` `q1` `q3`
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in Y axis.
Bars paddings in Bar charts.
Defines the inner & outer paddings for bars in Bar charts.
Inner padding between bars.
Outer padding for bars.
Chart description.
Chart footer text.
Chart size.
Chart width & height if customized, undefined if in "Fit to screen" mode.
string.
Values must be one of the following:
* `automatic`
Chart size width.
Chart size height.
Chart subtitle.
Chart title.
Slot value used to map color in the chart.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Categories of the color variable.
Each item in array.
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Defines if the discretization in the X axis uses a exact or pretty number of bins.
Defines if the discretization in the X axis uses quantiles.
Hides chart color legend title.
Hides chart X axis title.
Hides chart Y axis title.
Chart value labels.
Show & configure value labels in some charts.
Column for node labels in scatterplot charts.
Style configurations for value labels in bar charts.
Set position & alignment configuration for value labels in bar charts.
Value labels position in bar charts.
Values must be one of the following:
* `inside`
* `outside`
Value labels alignment in horizontal bar charts.
Values must be one of the following:
* `left`
* `middle`
* `right`
Format for value labels.
Format for absolute value labels.
Format for value count labels.
Format for percentage change labels.
Margin between node labels in scatterplot charts.
Whether to show value labels.
Whether to show absolute value labels.
Whether to show value count labels.
Whether to show percentage change labels.
Whether to show total value labels in stacked bar charts.
Metrics to show as value labels in boxplot charts.
Each item in array.
Chart color legend title.
Sorting applied to categories of the color variable (segmentsColumn).
Values must be one of the following:
`XAXIS` `YAXIS` `YAXIS_SEGMENT` `SELECTION` `ORDINAL` `ALPHANUM`
Sorting direction applied to categories of the color variable (segmentsColumn).
Values must be one of the following:
* `ASC`
* `DESC`
Sorting mode applied to categories of the color variable (segmentsColumn). CUSTOM does not apply any sorting criteria.
Values must be one of the following:
* `SORT`
* `CUSTOM`
Chart sorting applied to chart in the x axis.
Values must be one of the following:
`XAXIS` `YAXIS` `YAXIS_SEGMENT` `SELECTION` `ORDINAL` `ALPHANUM`
Chart sorting direction applied to chart in the x axis.
Values must be one of the following:
* `ASC`
* `DESC`
Stat configuration.
For example, for a max stat it would be:
`{ "stat": "max" }`
For stats that require a param like "countWhere" or "percentOfRowsWhere" it would be:
`{ "stat": "countWhere", "params": { "value": "categoryName" } }`
Stat kind.
Values must be one of the following:
`sum` `mean` `variance` `standardDeviation` `skewness` `kurtosis` `min` `p25` `p50` `p75` `max` `nNulls` `precision` `count` `cumSum` `nodeCount` `rForeground` `rForegroundColor` `rForegroundX` `uniqueValues` `valueCount` `countWhere` `percentOfRowsWhere`
Stat parameters.
Value for the stat parameter.
Theme applied to the chart.
Values must be one of the following:
`graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme color palette applied to the chart.
Values must be one of the following:
`Horus` `Osiris` `Re` `Blues` `Greens` `Oranges` `Purples` `Reds` `Viridis` `Magma` `Plasma` `Red-Blue` `Blue-Orange` `Red-Grey` `Red-Yellow-Blue` `Red-Yellow-Green` `graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme mode applied to the chart.
Values must be one of the following:
* `dark`
* `light`
Defines if tooltip is enabled.
Slot value represented in the x axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean, number or date, as well as listIndex.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
X Axis config options.
Configuration options specific to X axis, including base axis properties and label display controls.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis labels orientation mode.
Controls how axis labels are oriented. auto (rotates only when needed), rotate (always rotated), or none (never rotated).
Values must be one of the following:
* `auto`
* `rotate`
* `none`
Labels rotation in degrees.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart X axis title.
Slot value represented in the y axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean or number, as well as nodeCount or a metric.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Y Axis config options.
Configuration options specific to Y axis, including base axis properties.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart Y axis title.
Selection applied to chart in the x axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Selection applied to chart in the y axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Segmented Line Chart.
Segmented Line Chart.
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Defines if the cross-filters mode is set to auto.
Defines if the cross-filters and significant variables should ignore null values.
Insight default layout.
Values must be one of the following:
* `plot`
* `all`
Insight default layout elements.
Each item in array.
Values must be one of the following:
* `title`
* `description`
* `filters`
Defines if the cross-filters mode is set to relative.
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
A filter query.
See [Advanced query filters](https://docs.graphext.com/concepts/ds-concepts/filter-queries#advanced-filter-queries) for more information.
Title of the insight.
Comments that can be added to the chart.
Any text can be added as annotation at certain abscissa point.
Alignment of the annotation.
Values must be one of the following:
* `start`
* `center`
* `end`
Text of the comment.
Annotations shape.
Annotations shape configuration.
Annotation shape type.
Values must be one of the following:
* `arrow`
Annotation endpoint type for arrow shape.
Values must be one of the following:
* `line`
* `solid`
* `none`
Annotation shape line style.
Values must be one of the following:
* `solid`
* `dashed`
Annotation shape line color.
Annotation shape line dash.
Each item in array.
Annotation shape line width.
Annotation shape endpoint position x for arrow shape.
Annotation shape endpoint position y for arrow shape.
Annotations style.
Annotation label style configuration.
Annotation label color.
Annotation label font size in pixels.
Annotation label font style.
Values must be one of the following:
* `bold`
* `italic`
* `normal`
X axis value.
X axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in X axis.
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in X axis.
Y axis value.
Y axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Values must be one of the following:
`max` `mean` `median` `min` `q1` `q3`
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in Y axis.
Chart description.
Chart footer text.
Chart size.
Chart width & height if customized, undefined if in "Fit to screen" mode.
string.
Values must be one of the following:
* `automatic`
Chart size width.
Chart size height.
Chart subtitle.
Chart title.
Slot value used to map color in the chart.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Categories of the color variable.
Each item in array.
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Defines if the discretization in the X axis uses a exact or pretty number of bins.
Defines if the discretization in the X axis uses quantiles.
Hides chart color legend title.
Hides chart X axis title.
Hides chart Y axis title.
Interpolation method used for line or area chart.
Values must be one of the following:
`linear` `monotone` `cardinal` `natural` `step` `step-before` `step-after`
Chart value labels.
Show & configure value labels in some charts.
Column for node labels in scatterplot charts.
Style configurations for value labels in bar charts.
Set position & alignment configuration for value labels in bar charts.
Value labels position in bar charts.
Values must be one of the following:
* `inside`
* `outside`
Value labels alignment in horizontal bar charts.
Values must be one of the following:
* `left`
* `middle`
* `right`
Format for value labels.
Format for absolute value labels.
Format for value count labels.
Format for percentage change labels.
Margin between node labels in scatterplot charts.
Whether to show value labels.
Whether to show absolute value labels.
Whether to show value count labels.
Whether to show percentage change labels.
Whether to show total value labels in stacked bar charts.
Metrics to show as value labels in boxplot charts.
Each item in array.
Chart color legend title.
Line dash pattern.
Configures the dash pattern for lines in line charts.
Default dash pattern for lines.
It is represented as an array of stroke and gap lengths.
null.
array.
Each item in array.
Custom dash pattern.
Allows setting a specific dash pattern for the selected categories.
The custom dash pattern for the selected category.
It is represented as an array of stroke and gap lengths.
null.
array.
Each item in array.
Line markers.
Configures the markers for lines in line charts.
Line marker.
Configures a line marker.
The shape of the marker.
Values must be one of the following:
* `circle`
* `square`
* `triangle`
The color of the marker.
The area of the marker in pixels.
Custom line markers.
Allows setting a specific line marker for the selected categories.
Line marker.
Configures a line marker.
The shape of the marker.
Values must be one of the following:
* `circle`
* `square`
* `triangle`
The color of the marker.
The area of the marker in pixels.
Line width settings.
Specifies the width of the lines in line charts.
Default line width.
Custom line width.
Allows setting a specific line width for the selected categories.
The custom line width for the selected category.
Defines how to represent null values in line charts.
Values must be one of the following:
* `interpolated`
* `gap`
* `zero`
Sorting applied to categories of the color variable (segmentsColumn).
Values must be one of the following:
`XAXIS` `YAXIS` `YAXIS_SEGMENT` `SELECTION` `ORDINAL` `ALPHANUM`
Sorting direction applied to categories of the color variable (segmentsColumn).
Values must be one of the following:
* `ASC`
* `DESC`
Sorting mode applied to categories of the color variable (segmentsColumn). CUSTOM does not apply any sorting criteria.
Values must be one of the following:
* `SORT`
* `CUSTOM`
Stat configuration.
For example, for a max stat it would be:
`{ "stat": "max" }`
For stats that require a param like "countWhere" or "percentOfRowsWhere" it would be:
`{ "stat": "countWhere", "params": { "value": "categoryName" } }`
Stat kind.
Values must be one of the following:
`sum` `mean` `variance` `standardDeviation` `skewness` `kurtosis` `min` `p25` `p50` `p75` `max` `nNulls` `precision` `count` `cumSum` `nodeCount` `rForeground` `rForegroundColor` `rForegroundX` `uniqueValues` `valueCount` `countWhere` `percentOfRowsWhere`
Stat parameters.
Value for the stat parameter.
Theme applied to the chart.
Values must be one of the following:
`graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme color palette applied to the chart.
Values must be one of the following:
`Horus` `Osiris` `Re` `Blues` `Greens` `Oranges` `Purples` `Reds` `Viridis` `Magma` `Plasma` `Red-Blue` `Blue-Orange` `Red-Grey` `Red-Yellow-Blue` `Red-Yellow-Green` `graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme mode applied to the chart.
Values must be one of the following:
* `dark`
* `light`
Defines if tooltip is enabled.
Slot value represented in the x axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean, number or date, as well as listIndex.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
X Axis config options.
Configuration options specific to X axis, including base axis properties and label display controls.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis labels orientation mode.
Controls how axis labels are oriented. auto (rotates only when needed), rotate (always rotated), or none (never rotated).
Values must be one of the following:
* `auto`
* `rotate`
* `none`
Labels rotation in degrees.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart X axis title.
Slot value represented in the y axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean or number, as well as nodeCount or a metric.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Y Axis config options.
Configuration options specific to Y axis, including base axis properties.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart Y axis title.
Defines if zero values should be represented as null in line charts.
Selection applied to chart in the x axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Selection applied to chart in the y axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Stacked Area Chart.
Stacked Area Chart.
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Defines if the cross-filters mode is set to auto.
Defines if the cross-filters and significant variables should ignore null values.
Insight default layout.
Values must be one of the following:
* `plot`
* `all`
Insight default layout elements.
Each item in array.
Values must be one of the following:
* `title`
* `description`
* `filters`
Defines if the cross-filters mode is set to relative.
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
A filter query.
See [Advanced query filters](https://docs.graphext.com/concepts/ds-concepts/filter-queries#advanced-filter-queries) for more information.
Title of the insight.
Comments that can be added to the chart.
Any text can be added as annotation at certain abscissa point.
Alignment of the annotation.
Values must be one of the following:
* `start`
* `center`
* `end`
Text of the comment.
Annotations shape.
Annotations shape configuration.
Annotation shape type.
Values must be one of the following:
* `arrow`
Annotation endpoint type for arrow shape.
Values must be one of the following:
* `line`
* `solid`
* `none`
Annotation shape line style.
Values must be one of the following:
* `solid`
* `dashed`
Annotation shape line color.
Annotation shape line dash.
Each item in array.
Annotation shape line width.
Annotation shape endpoint position x for arrow shape.
Annotation shape endpoint position y for arrow shape.
Annotations style.
Annotation label style configuration.
Annotation label color.
Annotation label font size in pixels.
Annotation label font style.
Values must be one of the following:
* `bold`
* `italic`
* `normal`
X axis value.
X axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in X axis.
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in X axis.
Y axis value.
Y axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Values must be one of the following:
`max` `mean` `median` `min` `q1` `q3`
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in Y axis.
Chart description.
Chart footer text.
Chart size.
Chart width & height if customized, undefined if in "Fit to screen" mode.
string.
Values must be one of the following:
* `automatic`
Chart size width.
Chart size height.
Chart subtitle.
Chart title.
Slot value used to map color in the chart.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Categories of the color variable.
Each item in array.
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Defines if the discretization in the X axis uses a exact or pretty number of bins.
Defines if the discretization in the X axis uses quantiles.
Hides chart color legend title.
Hides chart X axis title.
Hides chart Y axis title.
Interpolation method used for line or area chart.
Values must be one of the following:
`linear` `monotone` `cardinal` `natural` `step` `step-before` `step-after`
Chart color legend title.
Sorting applied to categories of the color variable (segmentsColumn).
Values must be one of the following:
`XAXIS` `YAXIS` `YAXIS_SEGMENT` `SELECTION` `ORDINAL` `ALPHANUM`
Sorting direction applied to categories of the color variable (segmentsColumn).
Values must be one of the following:
* `ASC`
* `DESC`
Sorting mode applied to categories of the color variable (segmentsColumn). CUSTOM does not apply any sorting criteria.
Values must be one of the following:
* `SORT`
* `CUSTOM`
Stat configuration.
For example, for a max stat it would be:
`{ "stat": "max" }`
For stats that require a param like "countWhere" or "percentOfRowsWhere" it would be:
`{ "stat": "countWhere", "params": { "value": "categoryName" } }`
Stat kind.
Values must be one of the following:
`sum` `mean` `variance` `standardDeviation` `skewness` `kurtosis` `min` `p25` `p50` `p75` `max` `nNulls` `precision` `count` `cumSum` `nodeCount` `rForeground` `rForegroundColor` `rForegroundX` `uniqueValues` `valueCount` `countWhere` `percentOfRowsWhere`
Stat parameters.
Value for the stat parameter.
Theme applied to the chart.
Values must be one of the following:
`graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme color palette applied to the chart.
Values must be one of the following:
`Horus` `Osiris` `Re` `Blues` `Greens` `Oranges` `Purples` `Reds` `Viridis` `Magma` `Plasma` `Red-Blue` `Blue-Orange` `Red-Grey` `Red-Yellow-Blue` `Red-Yellow-Green` `graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme mode applied to the chart.
Values must be one of the following:
* `dark`
* `light`
Defines if tooltip is enabled.
Slot value represented in the x axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean, number or date, as well as listIndex.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
X Axis config options.
Configuration options specific to X axis, including base axis properties and label display controls.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis labels orientation mode.
Controls how axis labels are oriented. auto (rotates only when needed), rotate (always rotated), or none (never rotated).
Values must be one of the following:
* `auto`
* `rotate`
* `none`
Labels rotation in degrees.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart X axis title.
Slot value represented in the y axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean or number, as well as nodeCount or a metric.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Y Axis config options.
Configuration options specific to Y axis, including base axis properties.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart Y axis title.
Selection applied to chart in the x axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Selection applied to chart in the y axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Stacked Bar Chart.
Stacked Bar Chart.
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Defines if the cross-filters mode is set to auto.
Defines if the cross-filters and significant variables should ignore null values.
Insight default layout.
Values must be one of the following:
* `plot`
* `all`
Insight default layout elements.
Each item in array.
Values must be one of the following:
* `title`
* `description`
* `filters`
Defines if the cross-filters mode is set to relative.
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
A filter query.
See [Advanced query filters](https://docs.graphext.com/concepts/ds-concepts/filter-queries#advanced-filter-queries) for more information.
Title of the insight.
Comments that can be added to the chart.
Any text can be added as annotation at certain abscissa point.
Alignment of the annotation.
Values must be one of the following:
* `start`
* `center`
* `end`
Text of the comment.
Annotations shape.
Annotations shape configuration.
Annotation shape type.
Values must be one of the following:
* `arrow`
Annotation endpoint type for arrow shape.
Values must be one of the following:
* `line`
* `solid`
* `none`
Annotation shape line style.
Values must be one of the following:
* `solid`
* `dashed`
Annotation shape line color.
Annotation shape line dash.
Each item in array.
Annotation shape line width.
Annotation shape endpoint position x for arrow shape.
Annotation shape endpoint position y for arrow shape.
Annotations style.
Annotation label style configuration.
Annotation label color.
Annotation label font size in pixels.
Annotation label font style.
Values must be one of the following:
* `bold`
* `italic`
* `normal`
X axis value.
X axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in X axis.
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in X axis.
Y axis value.
Y axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Values must be one of the following:
`max` `mean` `median` `min` `q1` `q3`
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in Y axis.
Bars paddings in Bar charts.
Defines the inner & outer paddings for bars in Bar charts.
Inner padding between bars.
Outer padding for bars.
Chart description.
Defines the direction in which the chart is drawn.
Defines if Bar or BoxPlot charts will be drawn horizontally or vertically.
Values must be one of the following:
* `horizontal`
* `vertical`
Chart footer text.
Chart size.
Chart width & height if customized, undefined if in "Fit to screen" mode.
string.
Values must be one of the following:
* `automatic`
Chart size width.
Chart size height.
Chart subtitle.
Chart title.
Slot value used to map color in the chart.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Categories of the color variable.
Each item in array.
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Defines if the discretization in the X axis uses a exact or pretty number of bins.
Defines if the discretization in the X axis uses quantiles.
Hides chart color legend title.
Hides chart X axis title.
Hides chart Y axis title.
Chart value labels.
Show & configure value labels in some charts.
Column for node labels in scatterplot charts.
Style configurations for value labels in bar charts.
Set position & alignment configuration for value labels in bar charts.
Value labels position in bar charts.
Values must be one of the following:
* `inside`
* `outside`
Value labels alignment in horizontal bar charts.
Values must be one of the following:
* `left`
* `middle`
* `right`
Format for value labels.
Format for absolute value labels.
Format for value count labels.
Format for percentage change labels.
Margin between node labels in scatterplot charts.
Whether to show value labels.
Whether to show absolute value labels.
Whether to show value count labels.
Whether to show percentage change labels.
Whether to show total value labels in stacked bar charts.
Metrics to show as value labels in boxplot charts.
Each item in array.
Chart color legend title.
Sorting applied to categories of the color variable (segmentsColumn).
Values must be one of the following:
`XAXIS` `YAXIS` `YAXIS_SEGMENT` `SELECTION` `ORDINAL` `ALPHANUM`
Sorting direction applied to categories of the color variable (segmentsColumn).
Values must be one of the following:
* `ASC`
* `DESC`
Sorting mode applied to categories of the color variable (segmentsColumn). CUSTOM does not apply any sorting criteria.
Values must be one of the following:
* `SORT`
* `CUSTOM`
Chart sorting applied to chart in the x axis.
Values must be one of the following:
`XAXIS` `YAXIS` `YAXIS_SEGMENT` `SELECTION` `ORDINAL` `ALPHANUM`
Chart sorting direction applied to chart in the x axis.
Values must be one of the following:
* `ASC`
* `DESC`
Stat configuration.
For example, for a max stat it would be:
`{ "stat": "max" }`
For stats that require a param like "countWhere" or "percentOfRowsWhere" it would be:
`{ "stat": "countWhere", "params": { "value": "categoryName" } }`
Stat kind.
Values must be one of the following:
`sum` `mean` `variance` `standardDeviation` `skewness` `kurtosis` `min` `p25` `p50` `p75` `max` `nNulls` `precision` `count` `cumSum` `nodeCount` `rForeground` `rForegroundColor` `rForegroundX` `uniqueValues` `valueCount` `countWhere` `percentOfRowsWhere`
Stat parameters.
Value for the stat parameter.
Theme applied to the chart.
Values must be one of the following:
`graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme color palette applied to the chart.
Values must be one of the following:
`Horus` `Osiris` `Re` `Blues` `Greens` `Oranges` `Purples` `Reds` `Viridis` `Magma` `Plasma` `Red-Blue` `Blue-Orange` `Red-Grey` `Red-Yellow-Blue` `Red-Yellow-Green` `graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme mode applied to the chart.
Values must be one of the following:
* `dark`
* `light`
Defines if tooltip is enabled.
Slot value represented in the x axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean, number or date, as well as listIndex.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
X Axis config options.
Configuration options specific to X axis, including base axis properties and label display controls.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis labels orientation mode.
Controls how axis labels are oriented. auto (rotates only when needed), rotate (always rotated), or none (never rotated).
Values must be one of the following:
* `auto`
* `rotate`
* `none`
Labels rotation in degrees.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart X axis title.
Slot value represented in the y axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean or number, as well as nodeCount or a metric.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Y Axis config options.
Configuration options specific to Y axis, including base axis properties.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart Y axis title.
Selection applied to chart in the x axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Selection applied to chart in the y axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Stacked 100% Area Chart.
Stacked 100% Area Chart.
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Defines if the cross-filters mode is set to auto.
Defines if the cross-filters and significant variables should ignore null values.
Insight default layout.
Values must be one of the following:
* `plot`
* `all`
Insight default layout elements.
Each item in array.
Values must be one of the following:
* `title`
* `description`
* `filters`
Defines if the cross-filters mode is set to relative.
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
A filter query.
See [Advanced query filters](https://docs.graphext.com/concepts/ds-concepts/filter-queries#advanced-filter-queries) for more information.
Title of the insight.
Comments that can be added to the chart.
Any text can be added as annotation at certain abscissa point.
Alignment of the annotation.
Values must be one of the following:
* `start`
* `center`
* `end`
Text of the comment.
Annotations shape.
Annotations shape configuration.
Annotation shape type.
Values must be one of the following:
* `arrow`
Annotation endpoint type for arrow shape.
Values must be one of the following:
* `line`
* `solid`
* `none`
Annotation shape line style.
Values must be one of the following:
* `solid`
* `dashed`
Annotation shape line color.
Annotation shape line dash.
Each item in array.
Annotation shape line width.
Annotation shape endpoint position x for arrow shape.
Annotation shape endpoint position y for arrow shape.
Annotations style.
Annotation label style configuration.
Annotation label color.
Annotation label font size in pixels.
Annotation label font style.
Values must be one of the following:
* `bold`
* `italic`
* `normal`
X axis value.
X axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in X axis.
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in X axis.
Y axis value.
Y axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Values must be one of the following:
`max` `mean` `median` `min` `q1` `q3`
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in Y axis.
Chart description.
Chart footer text.
Chart size.
Chart width & height if customized, undefined if in "Fit to screen" mode.
string.
Values must be one of the following:
* `automatic`
Chart size width.
Chart size height.
Chart subtitle.
Chart title.
Slot value used to map color in the chart.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Categories of the color variable.
Each item in array.
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Defines if the discretization in the X axis uses a exact or pretty number of bins.
Defines if the discretization in the X axis uses quantiles.
Hides chart color legend title.
Hides chart X axis title.
Hides chart Y axis title.
Interpolation method used for line or area chart.
Values must be one of the following:
`linear` `monotone` `cardinal` `natural` `step` `step-before` `step-after`
Chart color legend title.
Sorting applied to categories of the color variable (segmentsColumn).
Values must be one of the following:
`XAXIS` `YAXIS` `YAXIS_SEGMENT` `SELECTION` `ORDINAL` `ALPHANUM`
Sorting direction applied to categories of the color variable (segmentsColumn).
Values must be one of the following:
* `ASC`
* `DESC`
Sorting mode applied to categories of the color variable (segmentsColumn). CUSTOM does not apply any sorting criteria.
Values must be one of the following:
* `SORT`
* `CUSTOM`
Stat configuration.
For example, for a max stat it would be:
`{ "stat": "max" }`
For stats that require a param like "countWhere" or "percentOfRowsWhere" it would be:
`{ "stat": "countWhere", "params": { "value": "categoryName" } }`
Stat kind.
Values must be one of the following:
`sum` `mean` `variance` `standardDeviation` `skewness` `kurtosis` `min` `p25` `p50` `p75` `max` `nNulls` `precision` `count` `cumSum` `nodeCount` `rForeground` `rForegroundColor` `rForegroundX` `uniqueValues` `valueCount` `countWhere` `percentOfRowsWhere`
Stat parameters.
Value for the stat parameter.
Theme applied to the chart.
Values must be one of the following:
`graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme color palette applied to the chart.
Values must be one of the following:
`Horus` `Osiris` `Re` `Blues` `Greens` `Oranges` `Purples` `Reds` `Viridis` `Magma` `Plasma` `Red-Blue` `Blue-Orange` `Red-Grey` `Red-Yellow-Blue` `Red-Yellow-Green` `graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme mode applied to the chart.
Values must be one of the following:
* `dark`
* `light`
Defines if tooltip is enabled.
Slot value represented in the x axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean, number or date, as well as listIndex.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
X Axis config options.
Configuration options specific to X axis, including base axis properties and label display controls.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis labels orientation mode.
Controls how axis labels are oriented. auto (rotates only when needed), rotate (always rotated), or none (never rotated).
Values must be one of the following:
* `auto`
* `rotate`
* `none`
Labels rotation in degrees.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart X axis title.
Slot value represented in the y axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean or number, as well as nodeCount or a metric.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Y Axis config options.
Configuration options specific to Y axis, including base axis properties.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart Y axis title.
Selection applied to chart in the x axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Selection applied to chart in the y axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Stacked 100% Bar Chart.
Stacked 100% Bar Chart.
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Defines if the cross-filters mode is set to auto.
Defines if the cross-filters and significant variables should ignore null values.
Insight default layout.
Values must be one of the following:
* `plot`
* `all`
Insight default layout elements.
Each item in array.
Values must be one of the following:
* `title`
* `description`
* `filters`
Defines if the cross-filters mode is set to relative.
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
A filter query.
See [Advanced query filters](https://docs.graphext.com/concepts/ds-concepts/filter-queries#advanced-filter-queries) for more information.
Title of the insight.
Comments that can be added to the chart.
Any text can be added as annotation at certain abscissa point.
Alignment of the annotation.
Values must be one of the following:
* `start`
* `center`
* `end`
Text of the comment.
Annotations shape.
Annotations shape configuration.
Annotation shape type.
Values must be one of the following:
* `arrow`
Annotation endpoint type for arrow shape.
Values must be one of the following:
* `line`
* `solid`
* `none`
Annotation shape line style.
Values must be one of the following:
* `solid`
* `dashed`
Annotation shape line color.
Annotation shape line dash.
Each item in array.
Annotation shape line width.
Annotation shape endpoint position x for arrow shape.
Annotation shape endpoint position y for arrow shape.
Annotations style.
Annotation label style configuration.
Annotation label color.
Annotation label font size in pixels.
Annotation label font style.
Values must be one of the following:
* `bold`
* `italic`
* `normal`
X axis value.
X axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in X axis.
Annotation value in X axis.
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in X axis.
Y axis value.
Y axis value where the annotation is placed.
Annotation kind.
Values must be one of the following:
* `VALUE`
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT_VALUE`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Annotation value in Y axis.
Annotation kind.
Values must be one of the following:
* `STAT`
Whether to hide the annotation value in the label.
Annotation stat in Y axis.
Values must be one of the following:
`max` `mean` `median` `min` `q1` `q3`
Annotation kind.
Values must be one of the following:
* `POSITION`
Annotation position in Y axis.
Bars paddings in Bar charts.
Defines the inner & outer paddings for bars in Bar charts.
Inner padding between bars.
Outer padding for bars.
Chart description.
Defines the direction in which the chart is drawn.
Defines if Bar or BoxPlot charts will be drawn horizontally or vertically.
Values must be one of the following:
* `horizontal`
* `vertical`
Chart footer text.
Chart size.
Chart width & height if customized, undefined if in "Fit to screen" mode.
string.
Values must be one of the following:
* `automatic`
Chart size width.
Chart size height.
Chart subtitle.
Chart title.
Slot value used to map color in the chart.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean or date.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Categories of the color variable.
Each item in array.
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Discretization configuration for a slot (x, y or color).
There is different kind of discretization configurations depending on column type:
* For quantitative ones a single count defining the number of bins is enough
* For categorical ones we need the count of categories & its top (DESC) or bottom (ASC) direction
* For dates we need a date period & a count of periods.
Number of bins in the axis.
Number of bins in the axis.
Get top (DESC) or bottom (ASC) categories for categorical variables in the axis.
Values must be one of the following:
* `ASC`
* `DESC`
Number of time units that sets the size of the discretization.
Time unit to specify the discretization period.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `DAY` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is a recurrent one (i.e. WEEK\_DAY).
Defines if the discretization in the X axis uses a exact or pretty number of bins.
Defines if the discretization in the X axis uses quantiles.
Hides chart color legend title.
Hides chart X axis title.
Hides chart Y axis title.
Chart value labels.
Show & configure value labels in some charts.
Column for node labels in scatterplot charts.
Style configurations for value labels in bar charts.
Set position & alignment configuration for value labels in bar charts.
Value labels position in bar charts.
Values must be one of the following:
* `inside`
* `outside`
Value labels alignment in horizontal bar charts.
Values must be one of the following:
* `left`
* `middle`
* `right`
Format for value labels.
Format for absolute value labels.
Format for value count labels.
Format for percentage change labels.
Margin between node labels in scatterplot charts.
Whether to show value labels.
Whether to show absolute value labels.
Whether to show value count labels.
Whether to show percentage change labels.
Whether to show total value labels in stacked bar charts.
Metrics to show as value labels in boxplot charts.
Each item in array.
Chart color legend title.
Sorting applied to categories of the color variable (segmentsColumn).
Values must be one of the following:
`XAXIS` `YAXIS` `YAXIS_SEGMENT` `SELECTION` `ORDINAL` `ALPHANUM`
Sorting direction applied to categories of the color variable (segmentsColumn).
Values must be one of the following:
* `ASC`
* `DESC`
Sorting mode applied to categories of the color variable (segmentsColumn). CUSTOM does not apply any sorting criteria.
Values must be one of the following:
* `SORT`
* `CUSTOM`
Chart sorting applied to chart in the x axis.
Values must be one of the following:
`XAXIS` `YAXIS` `YAXIS_SEGMENT` `SELECTION` `ORDINAL` `ALPHANUM`
Chart sorting direction applied to chart in the x axis.
Values must be one of the following:
* `ASC`
* `DESC`
Stat configuration.
For example, for a max stat it would be:
`{ "stat": "max" }`
For stats that require a param like "countWhere" or "percentOfRowsWhere" it would be:
`{ "stat": "countWhere", "params": { "value": "categoryName" } }`
Stat kind.
Values must be one of the following:
`sum` `mean` `variance` `standardDeviation` `skewness` `kurtosis` `min` `p25` `p50` `p75` `max` `nNulls` `precision` `count` `cumSum` `nodeCount` `rForeground` `rForegroundColor` `rForegroundX` `uniqueValues` `valueCount` `countWhere` `percentOfRowsWhere`
Stat parameters.
Value for the stat parameter.
Theme applied to the chart.
Values must be one of the following:
`graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme color palette applied to the chart.
Values must be one of the following:
`Horus` `Osiris` `Re` `Blues` `Greens` `Oranges` `Purples` `Reds` `Viridis` `Magma` `Plasma` `Red-Blue` `Blue-Orange` `Red-Grey` `Red-Yellow-Blue` `Red-Yellow-Green` `graphext` `dark` `excel` `fivethirtyeight` `ggplot2` `googlecharts` `latimes` `powerbi` `quartz` `urbaninstitute` `vox`
Theme mode applied to the chart.
Values must be one of the following:
* `dark`
* `light`
Defines if tooltip is enabled.
Slot value represented in the x axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean, number or date, as well as listIndex.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
X Axis config options.
Configuration options specific to X axis, including base axis properties and label display controls.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis labels orientation mode.
Controls how axis labels are oriented. auto (rotates only when needed), rotate (always rotated), or none (never rotated).
Values must be one of the following:
* `auto`
* `rotate`
* `none`
Labels rotation in degrees.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart X axis title.
Slot value represented in the y axis.
SlotValues could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts variables of type category, boolean or number, as well as nodeCount or a metric.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV`
Y Axis config options.
Configuration options specific to Y axis, including base axis properties.
Whether to hide grid lines in the axe or not.
Whether to include zero in the axe or not.
Whether to use a logarithmic scale.
Axis labels numeric format.
Maximum width in pixels for axis labels before truncating with ellipsis.
Axis ticks configuration.
An object for axis tick configuration.
Ticks count.
Equivalent to Vega axes tickCount prop.
number.
string.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Interval unit.
Values must be one of the following:
`millisecond` `second` `minute` `hour` `day` `week` `month` `year`
Step.
Ticks values.
Equivalent to Vega axes values prop.
Each item in array.
Chart Y axis title.
Selection applied to chart in the x axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Selection applied to chart in the y axis.
The chart will be zoom in over this selection.
array.
Each item in array.
array.
Each item in array.
Table Chart.
Table Chart.
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript
1 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
Theme mode applied to the whole insight.
Values must be one of the following:
* `dark`
* `light`
Defines if the cross-filters mode is set to auto.
Defines if the cross-filters and significant variables should ignore null values.
Insight default layout.
Values must be one of the following:
* `plot`
* `all`
Insight default layout elements.
Each item in array.
Values must be one of the following:
* `title`
* `description`
* `filters`
Defines if the cross-filters mode is set to relative.
Indicates if the insight can replay its original state or not.
Values must be one of the following:
* `drillDown`
* `directSelectionInGraph`
* `nonPersistedColumn`
* `True`
* `False`
A filter query.
See [Advanced query filters](https://docs.graphext.com/concepts/ds-concepts/filter-queries#advanced-filter-queries) for more information.
Title of the insight.
Defines the direction in which the chart is drawn.
Defines if Bar or BoxPlot charts will be drawn horizontally or vertically.
Values must be one of the following:
* `horizontal`
* `vertical`
Total number of selected rows in the dataset.
Values must be in the following range:
```javascript
0 ≤ selectedRows < inf
```
Width of the columns in the table.
Each object represents a column in the table and its corresponding width settings.
For example:
`{ "COLUMN-column1": 300 }`
Will set the width of the 'column1' to 300 pixels.
One or more additional parameters.
Discretizations for columns used in group by in the summary table.
Each object represents a dataset column and its corresponding discretization settings.
For example:
```json
{
"COLUMN-column1": { nBins: 10 }
}
```
Will try to discretize the 'column1' in 10 bins.
One or more additional parameters.
Configuration for the selected column.
Number of bins to discretize the column.
Values must be in the following range:
```javascript
0 ≤ count < inf
```
Period to discretize the column.
Values must be one of the following:
`MILLISECOND` `SECOND` `MINUTE` `HOUR` `YEAR_DAY` `MONTH_DAY` `WEEK_DAY` `WEEK` `WEEK_OF_YEAR` `MONTH` `QUARTER` `YEAR`
Defines if the period is recurrent.
Defines if the discretization for columns used in group by in the summary table uses exact or pretty number of bins.
Each object represents a dataset column and its corresponding configuration to use exact or pretty number of bins.
For example:
```json
{
"COLUMN-column1": true
}
```
Will try to discretize the 'column1' using an exact number of bins.
One or more additional parameters.
Defines if the discretization for columns used in group by in the summary table uses quantiles.
Each object represents a dataset column and its corresponding configuration to use quantiles.
For example:
```json
{
"COLUMN-column1": true
}
```
Will try to discretize the 'column1' using quantiles.
One or more additional parameters.
The index of the first row shown in the table.
Values must be in the following range:
```javascript
0 ≤ tableFirstIndex < inf
```
Format for the numeric values in the table.
Each object represents a column in the dataset and its corresponding format settings.
We use the Python format specification mini-language to specify the format.
For example:
```json
{ "COLUMN-column1": ",.2f" }
```
Will format the 'column1' with two decimal places (e.g., 1,234.56).
Each item in array.
Configuration for the selected column.
One or more additional parameters.
Whether to hide the index row in the table.
Variables to group by to generate a summary table.
If provided, tableValues should also be defined to specify the desired aggregation for each value.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts any variables from dataset.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV` `COUNT_WHERE` `PERCENT_OF_ROWS_WHERE`
Parameters for the aggregation method. Only for values with AGGREGATED\_COLUMN kind.
For example, for the aggregation 'COUNT\_WHERE', the parameter should include the category to count.
`{ "value": "category1" }`
Value for the aggregation method.
Whether the column is ordinal.
Custom name for the column.
Whether the label is custom.
If true, the label will be show without any transformation.
If false, the label may be transformed.
Whether to hide the column subtitle (e.g., `Value Count`).
Semantic type of the column.
Values must be one of the following:
`list` `boolean` `category` `currency` `date` `number` `sex` `text` `url` `list[boolean]` `list[date]` `list[number]` `list[url]` `list[category]`
Type of the column.
Values must be one of the following:
`categorical` `quantitative` `date` `text` `id` `url`
How many rows per page are shown in the table.
Values must be in the following range:
```javascript
1 ≤ tableRowsPerPage ≤ 100
```
Sorting criteria for a table.
Defines a column to sort the table with a specific sorting direction.
Sorting direction applied to the table.
Values must be one of the following:
* `ASC`
* `DESC`
Id of the table column to sort the table by.
Custom styles for the table.
Each item represents a column in the dataset and its corresponding style settings.
For example:
```json
[
{
"slotValueId": "COLUMN-column1",
"query": "FUZZY('positive')",
"style": {
"property": "graphextPill",
"value": "#FF0000"
}
}
]
```
Will apply a custom pill style with red color to the cells that match the query.
Slot value id.
Custom graphext query to apply the style.
Each cell will be evaluated with the query, and if it returns true, the style will be applied.
Categories for cell coloring.
Each item in the array represents a category in the column that will be used to color the cells in the table.
Each item in array.
Style configuration.
Each style is an object with property and value fields.
Style property to customize each cell:
* backgroundColor: will change the background color
* color: will change the text color
* graphextPill: will apply a custom pill style.
Values must be one of the following:
* `backgroundColor`
* `color`
* `graphextPill`
Color in hexadecimal format (e.g., `#FF0000`).
Totals to show as a footer in the table.
Each object represents a column in the dataset and the totals to show for the selected column
For example:
```json
{
"column1": ["sum", "mean", "variance"]
}
```
Will show the sum, mean and variance of the column1 (if column1 is shown in the table).
One or more additional parameters.
Totals to show for the selected column.
Each item in array.
Values must be one of the following:
`sum` `mean` `variance` `standardDeviation` `skewness` `kurtosis` `min` `p25` `p50` `p75` `max` `nNulls` `precision` `count` `mode` `median` `uniqueValues`
Aggregations if tableRows is provided, basic columns if not.
If tableRows is defined, tableValues will be the definition of desired aggregations. Each object represents a column in the dataset and its aggregation. Those aggregations will reduce all the values of the column to a single value, applying the selected aggregation method
If tableRows is not defined, tableValues will represent the columns to show in the table.
The value for a slot, defined by its kind & its name with some extra column props.
SlotValueData could be of kind column, aggregated column, listIndex, nodeCount or metric.
The kind of then slot value.
Values must be one of the following:
* `AGGREGATED_COLUMN`
* `COLUMN`
* `LIST_INDEX`
* `NODE_COUNT`
* `METRIC`
The name of the slot value.
Accepts any variables from dataset, as well as nodeCount or a metric.
Aggregation method to apply to the column. Only for values with AGGREGATED\_COLUMN kind.
Values must be one of the following:
`SUM` `AVG` `VARIANCE` `STDEV` `MIN` `P25` `P50` `P75` `MAX` `MODE` `UNIQUE_VALUES` `LIST` `LIST_UNIQUE` `CONCATENATE` `COUNT` `ELEMENT_COUNT` `ELEMENT_MIN` `ELEMENT_MAX` `ELEMENT_SUM` `ELEMENT_AVG` `ELEMENT_VARIANCE` `ELEMENT_STDEV` `COUNT_WHERE` `PERCENT_OF_ROWS_WHERE`
Parameters for the aggregation method. Only for values with AGGREGATED\_COLUMN kind.
For example, for the aggregation 'COUNT\_WHERE', the parameter should include the category to count.
`{ "value": "category1" }`
Value for the aggregation method.
Whether the column is ordinal.
Custom name for the column.
Whether the label is custom.
If true, the label will be show without any transformation.
If false, the label may be transformed.
Whether to hide the column subtitle (e.g., `Value Count`).
Semantic type of the column.
Values must be one of the following:
`list` `boolean` `category` `currency` `date` `number` `sex` `text` `url` `list[boolean]` `list[date]` `list[number]` `list[url]` `list[category]`
Type of the column.
Values must be one of the following:
`categorical` `quantitative` `date` `text` `id` `url`
# create_text_insight
Source: https://docs.graphext.com/api-docs/report/create_insight/create_text_insight
Create a new insight using only plain text.
## Usage
The following shows how the step can be used in a recipe.
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
create_text_insight(, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Title of the insight.
Type and appearance of an insight's elements.
A list in which each item is an object configuring the appearance of a particular insight element.
Specify the type of element.
Values must be one of the following:
`TITLE` `DESCRIPTION` `CHART` `GRAPH` `STATS` `LEGEND` `LEGEND_COLOR_SCALE` `TABLE_CHART`
The text shown for this insight element.
Element visibility.
The element's position and size.
The position refers to the top-left corner of the rectangle representing the insight element,
while the size is given by its width and height. For reference, the entire insight is 12 units
wide and 9 units high, and the origin (0, 0) of the x/y coordinates is in its top-left corner.
Horizontal position of the element's top-left corner.
In increments of 1; 0 being the left-most and 8 the right-most position.
Values must be in the following range:
```javascript theme={null}
-1 ≤ x ≤ 11
```
Vertical position of the element's top-left corner.
In increments of 1; 0 being the top-most and 11 the bottom-most position.
Values must be in the following range:
```javascript theme={null}
0 ≤ y ≤ 11
```
The width of the element (in increments of 1).
Values must be in the following range:
```javascript theme={null}
0 ≤ w ≤ 12
```
The height of the element (in increments of 1).
Values must be in the following range:
```javascript theme={null}
0 ≤ h ≤ 9
```
Name of the column containing the data to be used in this insight element.
*Required* if the element is of type CHART or STATS.
# Create Insight
Source: https://docs.graphext.com/api-docs/report/create_insight/index
| Step | Fast | Description |
| -------------------------------------------------------------------------------------------- | ---- | -------------------------------------------------- |
| [create\_compare\_insight](/api-docs/report/create_insight/create_compare_insight) | | Create a new insight from the Compare section |
| [create\_correlations\_insight](/api-docs/report/create_insight/create_correlations_insight) | | Create a new insight from the Correlations section |
| [create\_filter\_insight](/api-docs/report/create_insight/create_filter_insight) | | Create a new insight from a selection of nodes |
| [create\_graph\_insight](/api-docs/report/create_insight/create_graph_insight) | | Create a new insight from the Graph section |
| [create\_plot\_insight](/api-docs/report/create_insight/create_plot_insight) | | Create a new insight from the Plot section |
| [create\_text\_insight](/api-docs/report/create_insight/create_text_insight) | | Create a new insight using only plain text |
# create_project
Source: https://docs.graphext.com/api-docs/report/create_project/create_project
Prepare project using the final dataset.
## Usage
The following examples show how the step can be used in a recipe.
You can subset the columns to be included in the project in two ways. To explicitly include the columns you want to preserve using a list of their names:
```stan theme={null}
create_project(ds[["column_1", "keep_column_2"]])
```
Alternatively, explicitly exclude columns you are *not* interested in, using the negation "!":
```stan theme={null}
create_project(ds[!["column_3", "column_4"]])
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
create_project(ds: dataset)
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
A dataset containing the columns to be included in the project visualization.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
This step doesn't expect any configuration.
# Create Project
Source: https://docs.graphext.com/api-docs/report/create_project/index
| Step | Fast | Description |
| ----------------------------------------------------------------- | ---- | --------------------------------------- |
| [create\_project](/api-docs/report/create_project/create_project) | | Prepare project using the final dataset |
# export_to_airtable
Source: https://docs.graphext.com/api-docs/report/export/export_to_airtable
Export data to Airtable.
## Usage
The following examples show how the step can be used in a recipe.
Export dataset to an Airtable table
```stan theme={null}
export_to_airtable(ds, {"integration": "MY_AIRTABLE_INTEGRATION", "table_name": "Survey Results"})
```
Export and replace existing Airtable table data
```stan theme={null}
export_to_airtable(ds, {"integration": "MY_AIRTABLE_INTEGRATION", "table_name": "Customer Data", "if_exists": "replace"})
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
export_to_airtable(ds: dataset, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Dataset to be exported.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Identifier of the integration to use.
The id of the table you want to upload data to.
The preferred method for handling existing tables.
* 'Fail' if there is another table with the same name. The default value is 'fail' to prevent you from accidentally losing your data or compromising a table's structure in your database.
* 'Replace' if you want to override the existing table. Keep in mind this option deletes your previous data.
* 'Append' if you want to append the dataset's rows to the table.
Values must be one of the following:
* `fail`
* `replace`
* `append`
# export_to_amazonredshift
Source: https://docs.graphext.com/api-docs/report/export/export_to_amazonredshift
Export data to Amazon Redshift.
## Usage
The following examples show how the step can be used in a recipe.
Export dataset to an Amazon Redshift table
```stan theme={null}
export_to_amazonredshift(ds, {"integration": "MY_REDSHIFT_INTEGRATION", "table_name": "analytics_results"})
```
Append rows to an existing Redshift table
```stan theme={null}
export_to_amazonredshift(ds, {"integration": "MY_REDSHIFT_INTEGRATION", "table_name": "analytics_results", "if_exists": "append"})
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
export_to_amazonredshift(ds: dataset, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Dataset to be exported.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Identifier of the integration to use.
The name of the table you want to upload data to.
The preferred method for handling existing tables.
* 'Fail' if there is another table with the same name. The default value is 'fail' to prevent you from accidentally losing your data or compromising a table's structure in your database.
* 'Replace' if you want to override the existing table. Keep in mind this option deletes your previous data.
* 'Append' if you want to append the dataset's rows to the table.
Values must be one of the following:
* `fail`
* `replace`
* `append`
# export_to_amazons3
Source: https://docs.graphext.com/api-docs/report/export/export_to_amazons3
Export data to an AmazonS3 bucket.
## Usage
The following example shows how the step can be used in a recipe.
Export dataset to an Amazon S3 bucket
```stan theme={null}
export_to_amazons3(ds, {"integration": "MY_S3_INTEGRATION"})
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
export_to_amazons3(ds: dataset, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Dataset to be exported.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
ID of the integration you'd like to use.
# export_to_azureblob
Source: https://docs.graphext.com/api-docs/report/export/export_to_azureblob
Export data to an Azure Storage Blob.
## Usage
The following examples show how the step can be used in a recipe.
Export dataset to Azure Blob Storage
```stan theme={null}
export_to_azureblob(ds, {"integration": "MY_AZURE_BLOB_INTEGRATION", "blob_name": "exports/survey_results.csv"})
```
Export and overwrite an existing blob
```stan theme={null}
export_to_azureblob(ds, {"integration": "MY_AZURE_BLOB_INTEGRATION", "blob_name": "exports/latest_data.csv", "overwrite": true})
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
export_to_azureblob(ds: dataset, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Dataset to be exported.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
ID of the integration you'd like to use.
The name of the file you want to upload data to.
Configures this step's behaviour in case the file already exists.
# export_to_azuresql
Source: https://docs.graphext.com/api-docs/report/export/export_to_azuresql
Export data to Azure SQL.
## Usage
The following examples show how the step can be used in a recipe.
Export dataset to an Azure SQL table
```stan theme={null}
export_to_azuresql(ds, {"integration": "MY_AZURE_SQL_INTEGRATION", "table_name": "customer_segments"})
```
Replace an existing Azure SQL table
```stan theme={null}
export_to_azuresql(ds, {"integration": "MY_AZURE_SQL_INTEGRATION", "table_name": "customer_segments", "if_exists": "replace"})
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
export_to_azuresql(ds: dataset, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Dataset to be exported.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Identifier of the integration to use.
The name of the table you want to upload data to.
The preferred method for handling existing tables.
* 'Fail' if there is another table with the same name. The default value is 'fail' to prevent you from accidentally losing your data or compromising a table's structure in your database.
* 'Replace' if you want to override the existing table. Keep in mind this option deletes your previous data.
* 'Append' if you want to append the dataset's rows to the table.
Values must be one of the following:
* `fail`
* `replace`
* `append`
# export_to_bigquery
Source: https://docs.graphext.com/api-docs/report/export/export_to_bigquery
Export data to a BigQuery Table.
## Usage
The following examples show how the step can be used in a recipe.
Export dataset to a BigQuery table
```stan theme={null}
export_to_bigquery(ds, {"integration": "MY_BIGQUERY_INTEGRATION", "dataset_id": "my_dataset", "table_id": "survey_results"})
```
Append data to an existing BigQuery table
```stan theme={null}
export_to_bigquery(ds, {"integration": "MY_BIGQUERY_INTEGRATION", "dataset_id": "analytics", "table_id": "daily_metrics", "if_exists": "append"})
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
export_to_bigquery(ds: dataset, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Dataset to be exported.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
ID of the integration you'd like to use.
The name of the table you want to upload data to.
The name of the dataset you want to upload data to.
Values must match the following regex pattern:
```regex theme={null}
^[a-zA-Z0-9_]*$
```
The preferred method for handling existing tables.
* 'Fail' if there is another table with the same name. The default value is 'fail' to prevent you from accidentally losing your data or compromising a table's structure in your database.
* 'Replace' if you want to override the existing table. Keep in mind this option deletes your previous data.
* 'Append' if you want to append the dataset's rows to the table.
Values must be one of the following:
* `fail`
* `replace`
* `append`
# export_to_databricks
Source: https://docs.graphext.com/api-docs/report/export/export_to_databricks
Export data to Databricks.
## Usage
The following examples show how the step can be used in a recipe.
Export dataset to a Databricks table
```stan theme={null}
export_to_databricks(ds, {"integration": "MY_DATABRICKS_INTEGRATION", "database": "analytics_db", "table_name": "enriched_data"})
```
Replace an existing Databricks table
```stan theme={null}
export_to_databricks(ds, {"integration": "MY_DATABRICKS_INTEGRATION", "database": "analytics_db", "table_name": "enriched_data", "if_exists": "replace"})
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
export_to_databricks(ds: dataset, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Dataset to be exported.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Identifier of the integration to use.
The name of the table you want to upload data to.
The name of the database you want to upload data to.
The preferred method for handling existing tables.
* 'Fail' if there is another table with the same name. The default value is 'fail' to prevent you from accidentally losing your data or compromising a table's structure in your database.
* 'Replace' if you want to override the existing table. Keep in mind this option deletes your previous data.
* 'Append' if you want to append the dataset's rows to the table.
Values must be one of the following:
* `fail`
* `replace`
* `append`
# export_to_gdrive
Source: https://docs.graphext.com/api-docs/report/export/export_to_gdrive
Export data to a Google Drive file.
## Usage
The following example shows how the step can be used in a recipe.
Export dataset to Google Drive
```stan theme={null}
export_to_gdrive(ds, {"integration": "MY_GDRIVE_INTEGRATION"})
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
export_to_gdrive(ds: dataset, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
dataset.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
ID of the integration you'd like to use.
# export_to_gsheet
Source: https://docs.graphext.com/api-docs/report/export/export_to_gsheet
Export data to a Google Sheets sheet.
## Usage
The following example shows how the step can be used in a recipe.
Export dataset to a Google Sheets spreadsheet
```stan theme={null}
export_to_gsheet(ds, {"integration": "MY_GSHEETS_INTEGRATION"})
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
export_to_gsheet(ds: dataset, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
ID of the integration you'd like to use.
# export_to_notion
Source: https://docs.graphext.com/api-docs/report/export/export_to_notion
Export data to Notion.
## Usage
The following example shows how the step can be used in a recipe.
Export dataset to a Notion page
```stan theme={null}
export_to_notion(ds, {"integration": "MY_NOTION_INTEGRATION", "url": "https://www.notion.so/myworkspace/abc123def456"})
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
export_to_notion(ds: dataset, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Dataset to be exported.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Associated integration.
Notion Parent Page to write to.
We will put the table inside this page as a standalone page. Make sure the integration has the relevant permissions.
Values must match the following regex pattern:
```regex theme={null}
https:\/\/www\.notion\.so\/[a-zA-Z0-9_-]+\/[a-zA-Z0-9]+(\?[a-zA-Z0-9_=&-]*)?
```
# export_to_snowflake
Source: https://docs.graphext.com/api-docs/report/export/export_to_snowflake
Export data to Snowflake.
## Usage
The following examples show how the step can be used in a recipe.
Export dataset to a Snowflake table
```stan theme={null}
export_to_snowflake(ds, {"integration": "MY_SNOWFLAKE_INTEGRATION", "database": "ANALYTICS", "schema": "PUBLIC", "table_name": "survey_results"})
```
Append rows to an existing Snowflake table
```stan theme={null}
export_to_snowflake(ds, {"integration": "MY_SNOWFLAKE_INTEGRATION", "database": "ANALYTICS", "schema": "RAW", "table_name": "daily_imports", "if_exists": "append"})
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
export_to_snowflake(ds: dataset, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Dataset to be exported.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Identifier of the integration to use.
The name of the table you want to upload data to.
The name of the database you want to upload data to.
The name of the schema within the database you want to upload data to.
The preferred method for handling existing tables.
* 'Fail' if there is another table with the same name. The default value is 'fail' to prevent you from accidentally losing your data or compromising a table's structure in your database.
* 'Replace' if you want to override the existing table. Keep in mind this option deletes your previous data.
* 'Append' if you want to append the dataset's rows to the table.
Values must be one of the following:
* `fail`
* `replace`
* `append`
# export_to_sql
Source: https://docs.graphext.com/api-docs/report/export/export_to_sql
Export a given dataset to a specified SQL database.
This step uploads the given dataset to a table identified by the name provided in this step's parameters.
This table either exists already or it's going to be created in the SQL database configured in your SQL integration.
If you have several SQL integrations configured, this step will use the last one you added.
Further customization such as entry or table overwriting behavior can be achieved through parameters.
## Usage
The following examples show how the step can be used in a recipe.
Export dataset to a SQL database table
```stan theme={null}
export_to_sql(ds, {"integration": "MY_SQL_INTEGRATION", "table_name": "enriched_customers"})
```
Append rows to an existing SQL table
```stan theme={null}
export_to_sql(ds, {"integration": "MY_SQL_INTEGRATION", "table_name": "event_logs", "if_exists": "append"})
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
export_to_sql(ds: dataset, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Dataset to be uploaded.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
ID of the integration you'd like to use.
The name of the table you want to upload data to.
The preferred method for handling existing tables.
* 'Fail' if there is another table with the same name. The default value is 'fail' to prevent you from accidentally losing your data or compromising a table's structure in your database.
* 'Replace' if you want to override the existing table. Keep in mind this option deletes your previous data.
* 'Append' if you want to append the dataset's rows to the table.
Values must be one of the following:
* `fail`
* `replace`
* `append`
# export_to_tinybird
Source: https://docs.graphext.com/api-docs/report/export/export_to_tinybird
Export data to Tinybird.
## Usage
The following examples show how the step can be used in a recipe.
Export dataset to a Tinybird datasource
```stan theme={null}
export_to_tinybird(ds, {"integration": "MY_TINYBIRD_INTEGRATION", "datasource": "user_events"})
```
Replace an existing Tinybird datasource
```stan theme={null}
export_to_tinybird(ds, {"integration": "MY_TINYBIRD_INTEGRATION", "datasource": "user_events", "if_exists": "replace"})
```
General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.
```stan theme={null}
export_to_tinybird(ds: dataset, {
"param": value,
...
})
```
## Inputs & Outputs
The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).
Dataset to be exported.
## Configuration
The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.
Tinybird Integration to be used.
Data Source.
Configures this step's behaviour in case the datasource already exists.
If you want to override the existing datasource you can use `replace`, but keep in mind this deletes your previous data.
Otherwise if your desired outcome is to append the dataset's rows to the datasource you can use `append`.
The default value is set to `fail` to prevent you from accidentally losing data or compromising a datasource's structure.
Values must be one of the following:
* `fail`
* `replace`
* `append`
# Export
Source: https://docs.graphext.com/api-docs/report/export/index
| Step | Fast | Description |
| ------------------------------------------------------------------------------ | ---- | -------------------------------------------------- |
| [export\_to\_airtable](/api-docs/report/export/export_to_airtable) | | Export data to Airtable |
| [export\_to\_amazonredshift](/api-docs/report/export/export_to_amazonredshift) | | Export data to Amazon Redshift |
| [export\_to\_amazons3](/api-docs/report/export/export_to_amazons3) | | Export data to an AmazonS3 bucket |
| [export\_to\_azureblob](/api-docs/report/export/export_to_azureblob) | | Export data to an Azure Storage Blob |
| [export\_to\_azuresql](/api-docs/report/export/export_to_azuresql) | | Export data to Azure SQL |
| [export\_to\_bigquery](/api-docs/report/export/export_to_bigquery) | | Export data to a BigQuery Table |
| [export\_to\_databricks](/api-docs/report/export/export_to_databricks) | | Export data to Databricks |
| [export\_to\_gdrive](/api-docs/report/export/export_to_gdrive) | | Export data to a Google Drive file |
| [export\_to\_gsheet](/api-docs/report/export/export_to_gsheet) | | Export data to a Google Sheets sheet |
| [export\_to\_notion](/api-docs/report/export/export_to_notion) | | Export data to Notion |
| [export\_to\_snowflake](/api-docs/report/export/export_to_snowflake) | | Export data to Snowflake |
| [export\_to\_sql](/api-docs/report/export/export_to_sql) | | Export a given dataset to a specified SQL database |
| [export\_to\_tinybird](/api-docs/report/export/export_to_tinybird) | | Export data to Tinybird |
# Recipe steps
Source: https://docs.graphext.com/api-docs/steps
Working with data in Graphext's low-code mode
***
Writing a recipe is one way to instruct Graphext to build a project/visualization given some tabular data
(others being the use of the [Wizard](/concepts/graphext-concepts/wizard), or the application of pre-defined recipes).
A **recipe** itself is nothing more than a number of [steps](/concepts/graphext-concepts/steps), which are functions that accept some data and output new,
transformed or enriched data. A recipe can have an arbitrary number of such steps, and can generate an arbitrary
number of intermediate datasets. But, the output must always be a single dataset that serves as the basis for visual
exploration in the resulting project.
When you open the **Recipe Editor** for the first time in a newly created project, the inital dataset is made available
by default with the name `ds`, and so the simplest possible recipe is simply
```erlang theme={null}
create_project(ds)
```
i.e. a recipe with a single step called `create_project` which accepts a dataset as input and has no output. This is a
special case. Since the result of this step is the creation of a project, it doesn't generate any output that can be
further processed inside the recipe.
In practice you'll almost always want to somehow transform or enrich your dataset however, and so you'll want to add
one or more of the many steps available in Graphext before the final step of project creation.
# Steps
In general, the syntax for adding a step is very simple and always of the form:
```erlang theme={null}
step_name(inputs, ..., {params}) -> (outputs, ...)
```
i.e. you provide the name of the step, and in parentheses any inputs it will consume. The inputs may be either specific
columns of a dataset, a dataset itself, or a model. Details about the expected types of inputs depend on the specific
step in question, and will be documented in that step's page (see the categorized step documentation in the left sidebar).
As the last argument in parenthesis you can provide parameters to configure in detail how exactly the step will transform
the input data (if that step has such parameters of course; most but not all do). More on those below.
Finally, in another set of parenthesis (and separated by `->`), you provide names for the outputs that the step will generate.
Again, the outputs may be one more columns or datasets.
To differentiate between input datasets and columns, column names need to be prefixed with the name of the dataset it
belongs to, while datasets can be referred to by their name only. In other words, `ds` refers to the dataset with the name
"ds" and to pick out a specific column you'd use either `ds.my_column` or `ds["my_column"]`. The two forms are generally
interchangeable, but the latter is required if a column name contains spaces.
To given an example, a simple step that splits the texts in a given column in two at the first comma, might be written as
```erlang theme={null}
split_string(ds.text, {"pattern": ","}) -> (ds.left_part, ds.right_part)
```
The result of the split will be two new columns named "left\_part" and "right\_part" in the dataset "ds".
A very simple recipe including a transformation could thus be
```erlang theme={null}
split_string(ds.text, {"pattern": ","}) -> (ds.left_part, ds.right_part)
create_project(ds)
```
where the columns resulting from the split will now be included in the final project created.
Usually, when you start typing the beginning of a step's name in the **Recipe
Editor**, the rest of the step's signature will be autocompleted, including
the default names of any outputs it creates. So you only need to change the
names if you don't like the default ones (or if they clash with other outputs
you may have generated already).
# Parameters
Parameters let you configure how a step will process its inputs. The syntax of parameters corresponds to a valid json
object, for those familiar with json or javascript. For those who are not, it's simply a number of quoted parameter
names and corresponding values in between curly braces. E.g. we have already seen the example
```erlang theme={null}
{"pattern": ","}
```
where `"pattern"` is the parameter's name and `","` its value.
In general, all parameter names must be quoted strings, while values may be
* quoted strings
* numbers
* lists of numbers or strings
* another, nested object in curly braces, following the above rules
Each step's individual documentation will describe its valid parameters. And, even better, the recipe editor will help
you configure the step by highlighting any invalid parameters you may accidentally have selected.
# Classification model
Source: https://docs.graphext.com/concepts/ds-concepts/classification-model
Start exploring your data and discovering insights in under 5 minutes
# Correlation
Source: https://docs.graphext.com/concepts/ds-concepts/correlation
Start exploring your data and discovering insights in under 5 minutes
# Advanced Filter Queries
Source: https://docs.graphext.com/concepts/ds-concepts/filter-queries
Some steps allow the use of *advanced queries* to filter rows in a dataset. Right now, the following steps allow this:
* [create\_filter\_insight](https://docs.graphext.com/steps/report/create_insight/create_filter_insight/)
* [create\_graph\_insight](https://docs.graphext.com/steps/report/create_insight/create_graph_insight/)
* [create\_plot\_insight](https://docs.graphext.com/steps/report/create_insight/create_plot_insight/)
* [filter\_rows](https://docs.graphext.com/steps/prepare/filter/filter_rows/)
* [segment\_rows](https://docs.graphext.com/steps/prepare/transform/any/segment_rows/)
* [cluster\_network](https://docs.graphext.com/steps/analyse/graph_and_map/cluster/cluster_network/)
* [cluster\_subnetwork](https://docs.graphext.com/steps/analyse/graph_and_map/cluster/cluster_subnetwork/)
An *advanced query* allows you to filter data using a simple text query, the syntax of which may be familiar to users of SQL or Elasticsearch. For example, you may want to find the tweets with more than a certain amount of retweets - **"RT > 12"**; all customers that have paid their invoices - **"invoice\_paid: True"**; specific patterns in the title of documents - **"title: Obama"**; or find the votes whose values are greater than the median - **"num\_votes > MEDIAN"**.
*Advances queries* allow you to succinctly express such conditions, as well as compose multiple conditions in a single query.
## Text and categorical queries
In categorical or text columns you can search for specific words (tokens) or categories. You can combine searches using boolean logic (i.e. string together multiple conditions using the AND/OR keywords).
Select rows where the "department" column contains "engineering":
```
department: engineering
```
or where the "text" column contains "he" and "she":
```
text: he AND she
```
or filter the four most frequent categories in the "department" column:
```
department: TOP(4)
```
## Numerical and date queries
You can compare the values in numeric and date columns against constants to test for equality or against a desired range (mininum and/or maximum). For queries in date columns you can use dates expressed in the [ISO format 8061](https://en.wikipedia.org/wiki/ISO_8601) standard (e.g. "2019-01-01").
### Column statistics in filter conditions
In addition to using constants and ranges, you may also compare date and numeric values against certain column statistics. Specifically, all the following statistics are supported:
* **MIN**: minimum value
* **MAX**: maximum value
* **MEAN**: the average
* **P25**: or Q1, 25% of data lies below this point
* **MEDIAN**: that is, P50 or Q2. This is the median of the data and 50% of it lies below this point
* **P75**: or Q3, 75% of data lies below this point
### Date queries
Dates have to be specified in the [ISO format 8061](https://en.wikipedia.org/wiki/ISO_8601) standard, where the date part is required and the time part optional. I.e. both the following dates are valid: **2020-02-23** and **2020-02-23T14:30:00**. In the first example, only year, month and date are specified, while the second includes the time also (hours, minutes and seconds)
To select all rows whose "date" field falls into the year 2019:
```
date: >= 2019-01-01 AND <= 2019-12-31
```
Or to match a specific date:
```
date: 2019-01-01
```
The = operator is implicit in the above query, so the following query produces the same result:
```
date: =2019-01-01
```
Select dates before 2020, January 1st, 12:35 PM:
```
date: <= 2020-01-01T12:35:00
```
Dates before the median date in the same column:
```
date: >= 2020-02-23 AND <= MEDIAN
```
And you may of course also combine constants with calculated statistics:
```
date: >= 2020-02-23 AND <= P75
```
Note that queries in date columns only support **=**, **>=** or operators, while **>** and **\<** are not supported.
### Numerical queries
In numeric columns, numbers can also be specified using scientific notation. The following two numbers are both valid and represent the same number: **145000** and **1.45e5**.
To select rows whose age field falls into the range 10 to 55:
```
age: >= 10 AND <= 55
```
To match a specific number:
```
age: 12
```
The = operator is implicit in the above query, so the following query produces the same result:
```
age: =12
```
Include ages from 10 up to but not including 55 (exclusive smaller/greater than):
```
age: >= 10 AND < 55
```
Using scientific notation:
```
age: >= 5.6e-4 AND <= 9.35e-2
```
Using a computed statistic:
```
age: >= P25
```
Combining both constants and computed statistics:
```
age: >= 5 AND < MEAN
```
Queries in numeric columns support **=**, **>**, **>=**, **\<** and operators.
# Graphs and layouts
Source: https://docs.graphext.com/concepts/ds-concepts/graphs
One of Graphext’s most differentiating features is the way we allow you to see all your data at a glance in a single topological map, or graph. The Graph view intuitively highlights the local and global structures in your data by mapping rows that are similar to each other to close-by points on the screen, and conversely, mapping dissimilar rows to more distant points. This lets you immediately identify clusters of data points that are more similar amongst themselves than they are to others, detect outliers that are far away from the bulk of the data, and so on.
There are two principal approaches to construct such maps (also called “embeddings”) in Graphext:
1. through the creation of a *k-nearest neighbor graph* (k-NNG), followed by graph layout and clustering algorithms
2. through *dimensionality reduction* (DR) and non-graph based clustering algorithms
We will explain these two approaches in further detail in the following sections.
## TL;DR
Dimensionality reduction using UMAP is the faster and more precise method, and should be preferred unless the focus is on analysis of longish natural language texts. In the latter case, the k-NNG method currently provides a measure of text similarity that usually works a little better. Also, if the dataset contains a broad mix of quantitative and categorical variables, and UMAP dimensionality reduction does not generate the desired result, k-NNG may be worth trying, as it sometimes achieves a better balance between columns of different type.
## K-nearest neighbor graph
### Overview
The [k-NNG](https://en.wikipedia.org/wiki/Nearest_neighbor_graph) approach (k-Nearest Neighbor Graph) constructs a [graph](https://en.wikipedia.org/wiki/Graph_\(discrete_mathematics\)) (network) where each point in a dataset is connected to the k other points most similar to it. On its own, this simply constructs an *abstract* graph, consisting of *nodes* that represent the rows in the dataset, and *links* connecting them if they are similar enough.
In order to be able to view this graph on the (2-dimensional) screen, the nodes need to be assigned *coordinates* in the x/y plane. This is usually referred to as calculating a [*layout*](https://en.wikipedia.org/wiki/Graph_drawing) for the graph. The method employed in Graphext to do this is a [force-based layout](https://en.wikipedia.org/wiki/Force-directed_graph_drawing), which aims to place points such that the length of links reflects the similarity between them, while also aiming to avoid links crossing each other. We have chosen the [forceAtlas2](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0098679) layout algorithm as a sensible default, as it tends to work well enough across a variety of use cases.
The resulting graph, laid out on the 2-dimensional screen, can be thought of as an “embedding”. We have effectively embedded the original dataset in the space of x/y coordinates such that the positions of nodes reflect the similarity or dissimilarity between their corresponding dataset rows. While certain structures, like clusters of related points, are often immediately visible to the naked eye in such graphs, their identification can further be automated using clustering algorithms (in the context of graphs/networks also referred to as [community detection](https://en.wikipedia.org/wiki/Community_structure)). In Graphext, we use the [Louvain](https://en.wikipedia.org/wiki/Louvain_modularity) algorithm as a sensible default to detect clusters in graphs, as it produces good results in most cases. It also has the advantage of being fast to calculate, and so we can offer to re-calculate it on-the-fly from within Graphext’s UI (using the “Automatic segmentation” feature).
![K-NNG]()
### Technical details
#### Constructing the k-NN graph
Construction of a k-NN graph is simple if one is able to measure the similarity between pairs of data points. E.g. if a dataset only contains numeric columns, and each data point is thus a simple list of numbers (vector), we can simply use the [Euclidean distance](https://en.wikipedia.org/wiki/Euclidean_distance) to measure the similarity between two points (the straight-line distance between points in what we intuitively consider the normal 2d, 3d, or n-dimensional space, such as the x/y plane we know from school).
This approach is not applicable, however, if the dataset contains other types of data, such as categorical columns, dates, free-form natural language text and so on, since now we cannot easily interpret points as living in such a simple space (e.g. what would we be the coordinates of a categorical variable?). This means we need another measure of similarity that can be applied to datasets having mixed types of data. In Graphext, our choice is a custom measure that is somewhat similar to what’s known as the [Gower](https://www.jstor.org/stable/2528823) distance. The idea is simple: to measure the similarity between two data points, we apply a different measure for each type of column. E.g., for each numerical column we may apply some form of normalized difference between the quantitative values; for categorical columns we may simply check if the two points belong to the same category (distance = 0) or not (distance = 1); for text columns we may calculate similarity in terms of common word usage, etc. The final similarity then is simply the weighted average of all these individual column-wise measures. Since this provides a way to compare two points with arbitrary data types, we can then simply calculate the distances between all pairs, and use these to generate the k-NN graph.
#### Graph layout with forceAtlas2
We use our own (fast, C++) implementation of the force-based layout algorithm [forceAtlas2](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0098679). The basic idea of this algorithm is that nodes generally repel each other, while links act as springs applying an attracting force between connected nodes. The algorithm applies these forces to the nodes iteratively until a stable configuration is found in which the forces all balance out and the nodes stop moving (mostly). If the result is not ideal for a given dataset, e.g. nodes being too close together, or too far apart, [the algorithm has a number of parameters](https://pre.graphext.com/docs/steps/analyse/graph%20%26%20map/layout/layout_network/) that can be changed to create aesthetically different results (never changing the actual connectivity, and so also having no influence whatsoever on any clustering applied to the network, for example).
#### Graph clustering with Louvain
For graph clustering we also have our own, optimized implementation of the Louvain algorithm (written in Rust, and compiled to also work in the browser, i.e. from the Graphext UI). Louvain iteratively assigns nodes to clusters such as to maximize a measure called modularity, which is the relative density of edges within clusters with respect to edges between clusters. In other words, it tries to assign nodes to clusters such that there are many more connections between the nodes in the same cluster than there are between nodes in different clusters. As with the forceAtlas2 layout, the default configuration should generate decent results in most cases. However, a somewhat common occurrence is the generation of either too many or too few clusters. This can easily be addressed, however, as the algorithm has a resolution parameter that influences the number of clusters generated. For further details see the documentation of the corresponding [cluster\_network](https://pre.graphext.com/docs/steps/analyse/graph%20%26%20map/cluster/cluster_network/#parameters) step.
### Advantages & disadvantages
#### Advantages
* Works without having to preprocess mixed-type data, as we can use a different metric for each data type. This means less uncertainty about how the data is treated before any other algorithm is applied.
* Currently has the better metric for comparing longer texts (based on [Tf-Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf), rather than [word embeddings](https://en.wikipedia.org/wiki/Word_embedding) as used in the dimensionality reduction method explained below). Though this may change in the near future.
#### Disadvantages
* High complexity, i.e. slow execution time. The algorithm does not scale well, as the size of the dataset grows. Since we need to calculate the similarities between all pairs of dataset rows, the execution time scales quadratically with the number of rows. A dataset with twice the number of rows will take 4 times longer to process. More precisely, the complexity of the approach for *d* input dimensions and *n* rows scales as *O(d \* n^2)*.
* Doesn’t always work well with unusually distributed data (non-normal). For example, if a numerical column has a bi-modal distribution, i.e. with points being separated into two (or more) disparate regions, than (dis)similarity will be over-emphasized if two points fall into different regions, while similarity will be de-emphasized if two points fall within the same region. Effectively this means a loss of precision within each of the disparate regions, which in turn may result in a failure to separate some clusters.
### When to use
Since it doesn’t scale very well for larger datasets, we recommend the k-NNG method principally for analyses with a focus on longish natural language text, where the current Tf-Idf similarity tends to provide slightly better results than the word embeddings used in the dimensionality reduction approach. Also, if the dataset contains a broad mix of quantitative and categorical columns, and dimensionality reduction does not provide the desired result, it may be worth trying with k-NNG, as sometimes it achieves a better balance between columns of different data types.
### How to use in Graphext
#### Creating a k-NN graph in recipes
For each step in the k-NNG as described above, there is a corresponding method in Graphext to be used in recipes. A partial recipe may look like this, for example:
```python theme={null}
# Create network links calculating similarity between rows
link_similar_rows(ds[["content"]]) -> (links)
# Compute a force-directed graph layout with forceAtlas2
layout_network(links, {
"gravity": 0.03,
"avoidHubs": true,
"scalingRatio": 3
}) -> (ds.x, ds.y)
# Identify clusters in the network
cluster_network(links) -> (ds.cluster)
```
Here, the step [`link_similar_rows`](https://pre.graphext.com/docs/steps/analyse/graph%20%26%20map/create%20graph/link_similar_rows/) creates the abstract graph. Given an input dataset, the step’s output is a new dataset (`links`) in which each row is a link connecting one row to another. It has 3 columns: *source*, *target* and *weight*. The first two refer to the ids of the rows that are connected by the link (row numbers), while the *weight* column contains the numerical value of the similarity between the two rows.
To calculate node x/y positions we use the step [`layout_network`](https://pre.graphext.com/docs/steps/analyse/graph%20%26%20map/layout/layout_network/). This takes the just generated links dataset, and adds the *x* and *y* columns to the original rows, such that each row/node now has an associated position.
Finally, we detect clusters in the graph with [`cluster_network`](https://pre.graphext.com/docs/steps/analyse/graph%20%26%20map/cluster/cluster_network/). This step also receives the links dataset as input, and returns a simple categorical column containing the cluster labels.
#### Creating a k-NN graph using the Wizard
Details may vary depending on the version of Graphext (if used on-premise), and the particular choice of analysis selected initially in the Wizard, such as whether you’ve chosen to do a specific analysis of customers, say, or a generic segmentation (*Other*); but in most cases you should be able to select the option of doing a simple *Segmentation* (or *Segment your data*), which without further qualification means a clustering using the k-NNG approach (selecting *Segmentation using UMAP & HDBSCAN*, on the other hand, would correspond to the dimensionality reduction approach).
In newer versions of the Wizard, you may also find a *Network and Clusters* panel in the *Advanced Settings* section. Here you should be able to select *k-NNG & Louvain* as the Dimensionality reduction and clustering algorithms option. This will automatically configure the recipe to include the steps as shown above (or similar).
### Frequently asked questions
#### Why does the k-NNG approach not separate some of my cluster, when I think they should be separable?
* One reason may be that the data in the two supposed groups is really not different enough for any similarity metric to tell the difference.
* Another reason may be the loss of precision of this method with non-normal data, as mentioned in the *Disadvantages* section above. Check if any of your variables have a bi-modal distribution (with two distinct peaks instead of one), for example. If this is the case, you may want to try the dimensionality reduction approach instead.
* You may also find large undifferentiated “hairball” networks if the number of neighbors used to create the network is too large when compared to the size of the dataset. If you have a small dataset (say 100 rows), and you connect each row to a large number (say 20) of other rows, you may find that “everything becomes connected to everything”, and so there is no structure in the graph for the layout and clustering algorithms to work with. In this case try reducing the `n_similar_docs` parameter in [`link_similar_rows`](https://pre.graphext.com/docs/steps/analyse/graph%20%26%20map/create%20graph/link_similar_rows/).
#### My recipe seems to take a long time, how do I make it go faster?
As mentioned, the k-NNG approach is inherently slower than the dimensionality reduction described in the next section. The only way to reduce execution time somewhat is to use fewer variables, i.e. passing a smaller subset of columns to the [`link_similar_rows`](https://pre.graphext.com/docs/steps/analyse/graph%20%26%20map/create%20graph/link_similar_rows/) step.
## Dimensionality reduction
### Overview
#### Dimensionality reduction as layout
Our second approach to embedding data points is based on dimensionality reduction, and more specifically (at least for now), the [UMAP](https://umap-learn.readthedocs.io/en/latest/) algorithm. The idea, in this case, is to find a new way to represent our data using a smallish number of numerical columns (e.g. x and y only), but such that distances measured on our dimensionally reduced dataset maintain the same neighborhood structure as the original data. In other words, we want to find x/y positions for each data point, such that rows in the original dataset that are very similar will be found close to each other in terms of their x/y positions; and rows that are very dissimilar in the original dataset should be farther away from each other in terms of x and y. [UMAP](https://umap-learn.readthedocs.io/en/latest/) is a mathematically sound and principled approach to do just that.
Thus, when using dimensionality reduction (whether UMAP or other algorithms), we do not explicitly create a graph of nearest neighbors, and then calculate a layout. We rather consider the dimensionally reduced dataset itself our layout. Or, to state the same in reverse, to layout our dataset, we directly reduce our dataset to two dimensions (using UMAP or other algorithms) and interpret these as the data points’ x and y positions.
#### Dimensionality reduction for clustering
Note that the fewer dimensions we retain, the more information we lose from the original data. When calculating the 2-dimensional layout, the goal therefore is to at least capture the most salient structures in the dataset (e.g. clusters, the relative similarity between different clusters, local differences within clusters etc.). However, we do not have to restrict ourselves to always reducing our data to two dimensions only. For any purpose other than visualizing the dataset on the screen, it may make sense to keep more than two dimensions (and therefore more of the original information). This applies e.g. when trying to identify clusters. It may be, for example, that certain clusters can only be separated well in 3 dimensions (as an analogy, imagine e.g. one cloud behind another, and being able to distinguish them only from certain angles). In graphext, we therefore often embed our dataset twice using dimensionality reduction:
* Once, with exactly 2 dimensions for visualizing the dataset
* A second time, with about 10 dimensions for the purpose of clustering
Since the result of a dimensionality reduction is not a graph, but in essence simply a new dataset with fewer (numerical) columns, we cluster the reduced data not using graph-based community detection, but using “ordinary” clustering algorithms. By default this is [HDBSCAN](https://hdbscan.readthedocs.io/en/latest/how_hdbscan_works.html) in Graphext, but in principle it could also be k-means, agglomerative clustering etc. (see [scikit-learn](https://scikit-learn.org/stable/modules/clustering.html) for an overview of possibilities).
![DR]()
### Technical details
#### Preprocessing for UMAP
The vast majority of machine learning algorithms, including dimensionality reduction techniques like UMAP and cluster algorithms such as HDBSCAN, require their input data to be purely numerical and without any missing data. Few algorithms provide support for other data types (such as categorical, date and time etc.), for a mix of different types in the same dataset, or support missing data out of the box.
Graphext therefore provides a way to convert a dataset with arbitrary data types into a purely numerical dataset without missing data. It does so by defining for each possible type of input column a transformation from non-numeric to numeric values. As an example, ordered categorical variables (ordinals) such as the day of week, may be converted into a series of numbers (0..7). Non-ordered categorical variables of low-cardinality (containing few different categories) may be expanded into multiple new columns of 0s and 1s, indicating whether each row belongs to a specific category or not (one-hot encoding). Similar transformations are applied to dates, multivalued categoricals etc. Missing values (NaNs) are imputed, i.e. replaced with an appropriate value from the corresponding column (e.g. the median in a quantitative column). In addition, a new column of 0s and 1s is added, indicating whether the original column had a missing value or not.
All this is done automatically behind the scenes whenever you pass a dataset to an algorithm that expects its input to be numerical and complete. While this is convenient (since otherwise essentially no machine learning algorithm would be applicable to real-world datasets), it is currently not configurable, and so in specific, and hopefully rare, cases the way missing data is handled may not be ideal, for example. Note, however, that most of the time the machine learning algorithms we apply to your data are not supposed to give you the most precise predictions possible. Rather, the goal is to help you more easily explore your dataset and generate hypotheses (which you may then want to confirm or not using a more targeted approach).
#### Clustering using HDBSCAN
For many clustering algorithms it is difficult to determine in an intuitive way how many clusters should be found. In k-means, for example, you need to know beforehand how many clusters you’d like to identify, while many times such information is not available a priori. Other algorithms (such as Louvain, described above), may provide a resolution parameter, but again, it is not usually obvious how to select it. HDBSCAN tries to address this problem with a more intuitive configuration. Its main parameter influencing the resulting number of clusters is min\_cluster\_size, which as the name implies, is the minimum number of data points any group should have to be considered a cluster. Intuitively, HDBSCAN will (hierarchically) identify clusters of densely packed points, where groups smaller than min\_cluster\_size are considered noise if not part of a larger cluster in the hierarchy.
#### Adding explicit graph and Louvain clusters for UX consistency
As described above, although no explicit nearest neighbour graph is needed in principle to use the dimensionality reduction approach (neither for creating a layout, nor for clustering), Graphext will usually derive one anyway from the dimensionally reduced embeddings. Having the connectivity between data points available allows for easier navigation between neighbours in the UI, for example, and allows for the use of Louvain to cluster all or parts of the dataset on the fly (“automatic segmentation”). Note that the step to create this connectivity is not the same as the k-NNG approach however. Since we already have our dimensionally reduced embeddings (e.g. as the output of the embed\_dataset step), we can use very fast algorithms to find each row’s nearest neighbour (we use [Spotify’s Annoy](https://github.com/spotify/annoy) here), and so this extra step does not add significant computational overhead.
### Advantages & disadvantages
#### Advantages
* Unlike the k-NNG approach, it doesn’t make any assumptions about the distributions of input variables. It therefore also doesn’t suffer from a loss of “precision” in the case of non-normal data.
* It scales better with increasing number of rows. Behind the scenes, the algorithm doesn’t need to calculate all pairwise similarities, but uses an approximation instead to find nearest neighbors. This means it doesn’t scale quadratically (the approximate complexity seems to be around *O(d\*n^1.14)* for *d* input dimensions and *n* rows, also see [here](https://github.com/lmcinnes/umap/issues/8)).
* It is a [published](https://arxiv.org/abs/1802.03426) algorithm with strong mathematical justification and proven usage in various fields (single cell analysis, neural network activations, time series analysis [and more](https://umap-learn.readthedocs.io/en/latest/scientific_papers.html)).
#### Disadvantages
* As explained in Technical Details, the UMAP approach relies on inputs being numerical, which means we need to first transform any non-numerical data before applying the algorithm. Since this is done behind the scenes (for simplicity on the user side), and currently not configurable, this leads to less transparency in what data is eventually processed during dimensionality reduction and clustering.
* To be able to provide dataset “navigability” in Graphext, i.e. being able to select a node’s neighbors etc., an explicit k-NN graph is eventually constructed anyways, although not in principle necessary for neither dimensionality reduction nor clustering. Though fast in terms of execution, this leads to somewhat more complexity in recipes using this approach (also see Fig. 2).
### When to use
Ignoring recipes provided out-of-the box in Graphext, we recommend using the dimensionality reduction approach with UMAP as default in most cases, more so when datasets are large, or when little is known about the distributions in the data. UMAP is considerably faster, and generally more precise in maintaining both local and global similarity between data points. When analyzing longish texts, however, the text similarity measure used by the k-NNG approach works a little better sometimes. Also, when working with a lot of mixed variables types (numerical, categorical etc.), the k-NNG may be worth trying, since in some cases it seems to achieve a better balance between the different types.
### How to use in Graphext
#### Performing a dimensionality reduction in recipes
Again, each step in the flowchart for dimensionality reduction corresponds to a step/method in Graphext recipes. For example:
```python theme={null}
# Reduce dataset to 2 dimensions (x/y node positions)
layout_dataset(ds) -> (ds.x, ds.y)
# Reduce the dataset to a 10-dimensional numeric vector embedding
embed_dataset(ds) -> (ds.embedding)
# Identify clusters using the distance between provided embeddings
cluster_embeddings(ds.embeddings) -> (ds.cluster)
# Create network links by calculating the similarity of embeddings
link_embeddings(ds.embedding) -> (links)
```
Here, [`layout_network`](https://pre.graphext.com/docs/steps/analyse/graph%20%26%20map/layout/layout_network/) is used to reduce the input dataset to 2 dimensions (x/y) using UMAP. Next, [`embed_dataset`](https://pre.graphext.com/docs/steps/prepare/embed/embed_dataset/) employs UMAP again to create 10-dimensional embeddings. In the resulting output column, each original dataset row is now represented by a vector of 10 numbers.
Having embedded the dataset in 10 dimensions, we then use [`cluster_embeddings`](https://pre.graphext.com/docs/steps/analyse/graph%20%26%20map/cluster/cluster_embeddings/) with the HDBSCAN algorithm by default to identify clusters in the dataset. Finally, [`link_embeddings`](https://pre.graphext.com/docs/steps/analyse/graph%20%26%20map/create%20graph/link_embeddings/) is used to calculate for each embedded row its k nearest neighbours. The output is again a dataset of links indicating which nodes should be connected as well as how similar they are quantitatively.
#### Performing a dimensionality reduction using the Wizard
Again, details may vary depending on the version of Graphext (if used on-premise), and the particular choice of analysis selected initially in the Wizard; but if the type of analysis supports the use of dimensionality reduction, then you should be able to select something like *Segmentation using UMAP & HDBSCAN* as an option from the first set of questions (if the segmentation option doesn’t specify any concrete method, the k-NNG approach is used instead).
In newer versions of Graphext, you may also find a *Network and Clusters* panel in the *Advanced Settings* section of the Wizard. Here you should be able to select *UMAP & HDBSCAN* as the *Dimensionality reduction and clustering algorithms* option. This will automatically configure the recipe to include the steps as shown above (or similar).
### Frequently asked questions
#### UMAP (embed\_dataset) is still taking a long time, how can I accelerate its execution?
Although UMAP is already [pretty fast](https://umap-learn.readthedocs.io/en/latest/performance.html) (when compare to related algorithms like t-SNE), for very large dataset there are a number of tweaks that increase performance at the cost of a possible (but not necessary) reduction in precision:
* *random\_state*: if determinism is not required, the parameter `random_state` can be set to null (`{“random_state”: null}`). This will enable parallel execution of the UMAP algorithm. This is still an experimental feature and so may or may not result in faster performance. It will, mean, however, that repeated executions of the step, even with identical input data, may produce different results. Note that these differences are usually negligible, i.e. they will not change any of the general structures inherent in the data, and therefore won’t usually affect any insights that can be derived from it.
* *init*: by default, before applying its own logic to the data, UMAP initializes the layout of data points using the “spectral” method. For large dataset this can be somewhat costly. In this case you may also try with `{“init”: null}`, which should be faster, at the cost of potentially being less precise.
* *n\_neighbors*: this parameter controls how many nearest neighbors UMAP takes into account when calculating the layout/embedding. The smaller the number, the faster the execution and the greater the focus on the local structure of the data. The greater the number, the slower the execution and the greater the focus on the larger-scale structures in the data.
#### Some clusters seem very far apart. How can I avoid large distances between clusters?
In some cases, if clusters are sufficiently different from each other (if the corresponding graph is essentially disconnected), UMAP may push the clusters rather far apart from each other. Visually this means individual points may become small in the UI, and although this doesn’t affect the “quality” of the embedding, it may make navigating the resulting layout less convenient. In this case you may try to change the parameter `n_epochs` (which is 200 by default for large datasets and 500 for smaller ones). Setting this to smaller values means the algorithm will have less time to push clusters apart, but also less time to converge on the “optimal” solution.
#### I see strange long threads of data points in the UMAP layout, instead of more patch-like clusters. What happened?
This is usually a sign that you’ve used only (or principally) input columns that are (nearly) unique, i.e. have a different value in each row. Typical cases are e.g. the use of numeric customer IDs or dates, where each row has a different ID or date. If this is the only information the algorithm has to go on, then each row will essentially be connected to the next higher and lower ID or date, and the result will be a more or less tangled thread of data points with values increasing or decreasing from one end of the thread to the other. Simply include more informative features in the corresponding steps to create a proper clustering.
## But does it work?
Since the embedding of a dataset, and its clustering, is usually done unsupervised, i.e. without knowing the ground truth for at least part of the data, there is no direct way to confirm whether a particular clustering is “correct”, or even any “good” or “bad”. In fact, without a ground truth, or some other context, any clustering whatsoever is as valid as any other. What matters in this case, usually, is whether the clustering is useful. Being useful here could mean, e.g., that different clusters of customers correspond to my intuition about the groups of customers I expect; or that the clusters allow me to predict, understand or classify customer behaviour more easily when compared to not distinguishing between groups of them.
### Validation with labelled data
Fortunately, for the purpose of evaluating a particular approach or algorithm, we can make use of labelled data (where the ground truth is known) to get an idea of its usefulness. The idea here is simply to see whether the clusters identified automatically and from the features of the dataset alone, correspond to the labels a human has assigned to data points manually. E.g., the famous [Iris dataset](https://en.wikipedia.org/wiki/Iris_flower_data_set), contains information about 3 different species of Iris flowers. It provides 4 variables that describe the shape of their petals and stalks, as well as a 5th column indicating the species (versicolor, setosa or virginica). To see whether any of Graphext’s embedding and clustering approach actually “works”, we simply embed and cluster the dataset using the four feature variables only, and then compare whether the found clusters of flowers correspond to the different species as identified in the label column. In other words, we’d expect all flowers in the same, automatically identified, cluster to be of the same species, even though we haven’t used information about species in the clustering.
Of course, this will only work if the features provided in the dataset correlate to a sufficient degree with the labels. If the measured features of the flowers didn’t tell us anything about their species, then a clustering wouldn’t be able to “recover” the species either. Fortunately, there are many datasets used to benchmark classification algorithms, which we can simply re-purpose for the task of cluster evaluation. Doing this for a sufficiently diverse number of datasets (in terms of size, complexity, data types etc.), we can convince ourselves that, and to which degree, each approach recovers the original labels.
We include and share a number of projects doing just this in the Graphext team [Validation](https://app.graphext.com/teams/VGVhbS0yMDEw/projects/) (you may need to ask us to add you to the team to be able to access it). All projects here are based on typical datasets used to benchmark classification algorithms. As such, each contains the human-annotated ground-truth labels in a categorical column, which we’ve pinned to the top of the left column in the Graphext UI. Just below on the left, you’ll then find the clusters we’ve identified without taking into account the true labels.
To easily compare the two, you can e.g. color the nodes by the true labels and confirm that each true class corresponds to one (or more) clusters (sometimes automatic clusters will make a finer or coarser distinction between groups of similar data points). E.g., in the [Iris project](https://app.graphext.com/projects/UHJvamVjdC0yMTMxOQ==/v/graph?colorMap=species\&areaMap=null), with color and node title indicating the true species, the following figure confirms that automatic clusters almost perfectly distinguish the 3 different species:
![iris dataset]()
Conversely, you could select each automatic cluster and confirm the “purity” of the true labels of the corresponding nodes. In Figure 4, for example, we have selected cluster 2, with true labels being \~99% “versicolor” and a single node corresponding to an unusually small example of virginica; thus confirming that the cluster (like the others) does indeed correspond to a particular species of flower.
![iris dataset]()
### Example validation projects
We have included a variety of different datasets and projects in the [Validation](https://app.graphext.com/teams/VGVhbS0yMDEw/projects/) team. For each dataset we have created at least one project using the UMAP approach, and one using the k-NNG approach, with the name indicating which was used. The ground truth labels are always to be found on the top-left, followed by the automatic clusters. See the box below for a list of current validation projects (at the time of writing).
**Small and mostly (or only) numerical data**
* Iris ([Scikit-learn dataset](https://scikit-learn.org/stable/datasets/index.html#iris-plants-dataset))
150 rows, 4 numeric features. Classes: 3 species of flowers.
* Penguins ([Dataset on github](https://github.com/allisonhorst/palmerpenguins#palmerpenguins-))
344 rows, uses the 4 numeric features. Classes: 3 species of penguins.
* Wine ([Scikit-learn dataset](https://scikit-learn.org/stable/datasets/index.html#wine-recognition-dataset))
178 rows, 13 numeric features. Classes: 3 types of wine.
* Breast cancer ([Scikit-learn dataset](https://scikit-learn.org/stable/datasets/index.html#breast-cancer-wisconsin-diagnostic-dataset))
569 rows, 30 numeric features. Classes: 2 (malign, benign)
**Mostly categorical data**
* Mushrooms ([UCI dataset](https://archive.ics.uci.edu/ml/datasets/Mushroom))
8124 rows, 21 categorical features, 1 numerical. Classes: 2 (edible, poisonous)
**Mixed data types**
* Adults ([UCI dataset](http://archive.ics.uci.edu/ml/datasets/Adult))
48,842 rows, 6 numeric features, 9 categorical. Classes: 2 incomes (\<= 50k, >50k)
**Image data**
* Nist Digits ([Scikit-learn dataset](https://scikit-learn.org/stable/datasets/index.html#optical-recognition-of-handwritten-digits-dataset))
5,620 handwritten digits, each image containing 8x8=64 pixels. 10 classes (0, 1, ..., 9).
* Fashion Mnist ([Dataset on Github](https://github.com/zalandoresearch/fashion-mnist#fashion-mnist))
70,000 images of clothing items, 28x28 pixels. Classes: 10.
**Text data**
* News ([Scikit-learn dataset](https://scikit-learn.org/stable/datasets/index.html#the-20-newsgroups-text-dataset))
18,846 newsgroup posts (message body only). Classes: 20 (newsgroup topics).
Note that depending on the size of the dataset we may have adjusted a few parameters in the projects’
recipes when compared to their defaults (usually considering fewer/more neighbouring nodes in an algorithm,
or adjusting the minimum size of clusters). Also, when processing texts or images, for example, there may be
slight variations from the example recipes described above. You can always select “Recreate project” from the
sandwich menu in the top left corner of a project view to inspect the recipe and how each algorithm was configured.
## Summary
We have given an overview of the two principal approaches to laying out a dataset in Graphext:
* Building a k-nearest neighbor graph and applying graph-based clustering ([Louvain](https://en.wikipedia.org/wiki/Louvain_modularity)). The recipe steps involved here are [link\_similar\_rows](https://pre.graphext.com/docs/steps/analyse/graph%20%26%20map/create%20graph/link_similar_rows/), [layout\_network](https://pre.graphext.com/docs/steps/analyse/graph%20%26%20map/layout/layout_network/), and [cluster\_network](https://pre.graphext.com/docs/steps/analyse/graph%20%26%20map/cluster/cluster_network/).
* Using dimensionality reduction ([UMAP](https://umap-learn.readthedocs.io/en/latest/)) and density-based clustering ([HDBSCAN](https://hdbscan.readthedocs.io/en/latest/index.html)). The recipes steps involved here are primarily [layout\_dataset](https://pre.graphext.com/docs/steps/analyse/graph%20%26%20map/layout/layout_dataset/), [embed\_dataset](https://pre.graphext.com/docs/steps/prepare/embed/embed_dataset/), and [cluster\_embeddings](https://pre.graphext.com/docs/steps/analyse/graph%20%26%20map/cluster/cluster_embeddings/).
We have compared the advantages and disadvantages of each approach and explained when to use them. Finally, we have described a method to evaluate how well these methods work using human-annotated datasets, in which the ground truth can be compared with automatically identified clusters. A number of such datasets and corresponding projects using each approach are publicly available in Graphext.
# Graphs and data types
Source: https://docs.graphext.com/concepts/ds-concepts/graphs-and-data-types
## Introduction
[*UMAP*](/guides/graphs/#dimensionality-reduction-as-layout) and our proprietary [*k-NN*](http://127.0.0.1:8000/guides/graphs/#overview) graph treat the relative balance between categorical and numeric data differently, leading most of the time to qualitatively different embeddings. The k-NNG method has a tendency to clearly separate the network into clusters such that each cluster corresponds to a combination of categories (across different variables). UMAP tends to not divide the network as sharply based on categorical columns, yet can be configured to give them more or less influence on the resulting embedding.
## Example 1 – Human resources data
As a first example, we use a human resources dataset with 10 numeric and 2 categorical columns. The categorical columns represent the *salary* of employees (3 levels) and their *department* (10 levels). We will also be showing two numeric columns: the *number of projects* (integer values from 2 to 7), and *satisfaction level* (continuous in \[0, 1]).
### k-NNG embedding
The k-NNG method produces the following layout:
![graph layout]()
![graph layout]()
![graph layout]()
![graph layout]()
Coloring in the top row by salary on the left, and department on the right, we see that the network forms clearly distinct clusters, and such that each cluster corresponds to a combination of the two kinds of categories. This is neither right nor wrong. And whether it's useful depends on the kinds of questions we're interested in. E.g. separating employees so clearly by department may or may not make sense in a specific scenario, since belonging to a specific department may or may not have a significance influence on other variables we may be interested in.
Within each cluster, points are organized according to the numeric variables, forming different directions/gradients of increasing or decreasing values etc. This is more visible in case of the ordinal-like variable *number of projects,* having 7 different values only, than in the case of *satisfaction level*.
### UMAP embedding
Using UMAP has the advantage of giving us more influence on the relative importance of categorical vs numeric variables. In Graphext, this can be done with the `type_weights` parameter, e.g.
```jsx theme={null}
embed_dataset(ds, {"type_weights": {"category": 4.0}}) -> (ds.embedding)
```
In the following series of figures we plot UMAP layouts with different weights being applied to categorical columns.
With default weighting, and using the same color mappings as before:
![graph layout]()
![graph layout]()
![graph layout]()
![graph layout]()
It is clear that UMAP in this case globally organizes the data according to similarity in the numeric columns, more so than according to the categorical columns.
Note, however, that *salary* data is not randomly or uniformly distributed here, even if that may seem to be the case at first glance. Even though the different levels of the categorical variables can be found spread across the entire network, they nevertheless tend to form little groups, i.e. nodes of the same category more often connect to each other than do nodes with different categories.
Increasing the weight of categorical columns to 6.0 (vs. the default of 1.0 for the rest), we see that data starts to separate into clusters representing different *salary* categories (top left e.g.).
![graph layout]()
![graph layout]()
With a weight of 8, the salary variable starts to dominate the global structure, and the network separates into 3 distinct clusters. The department variable also starts to separate parts of the network more clearly:
![graph layout]()
![graph layout]()
The same tendency continues with a weight of 10.
![graph layout]()
![graph layout]()
And with weights of 12 and 14 respectively, the network separates into completely distinct clusters, such that each cluster corresponds to a combination of the categories in the two categorical variables.
![graph layout]()
![graph layout]()
![graph layout]()
![graph layout]()
Note that in no case are numeric columns ignored, even when the network separates into disconnected islands representing individual categories. The following figures again map color by our two numeric columns, using an embedding with categorical weight of 14:
![graph layout]()
![graph layout]()
As can be seen, local structure in numeric variables is preserved inside and/or across the individual clusters. If anything, in a more organized manner than the original k-NN graph.
Note that apart from the above observations, UMAP also better maintains the relationship *between* clusters. While the relative positions of isolated clusters in the k-NN graph are completely arbitrary (as a direct implication of the underlying algorithm), UMAP is able to maintain the global structure inherent in the data.
## Example 2 – Titanic dataset
As a another example we will look at (a subset of) the Titanic dataset, which for each passenger contains 3 categorical columns (*sex*, *passenger class,* and *port of embarkment*), and 4 numeric columns (*age*, *number of siblings and spouses aboard*, *number of parents and children aboard*, and *fare price*).
### k-NNG embedding
The k-NNG method produces the following layout (*double-click on an image to see a bigger version*):
![graph layout]()
![graph layout]()
![graph layout]()
![graph layout]()
As before we can see that the k-NNG method strongly separates different categories into separate clusters. Numeric variables than usually influence how nodes are distributed *inside* the clusters. Note that the approach seems to struggle somewhat in mapping the *age* variable\*.\*
### UMAP embedding
For this dataset, using UMAP with default column weights seems to achieve a good balance between categorical and numerical variables out of the box (note that in the following figures we changed the `min_dist`, and `n_epochs` parameters only, as with default values the resulting layout is rather thinly spread):
![graph layout]()
![graph layout]()
![graph layout]()
![graph layout]()
Note that not only are the different categories well represented by identifiable clusters, but numeric variables map much more cleanly to well defined areas or gradients in the layout, when compared with the k-NNG approach.
## Conclusions
As we have illustrated with two examples above, it is possible in many cases to tune UMAP embeddings such that the equilibrium between categorical and numerical variables corresponds to one's expectations (e.g. qualitatively matches results from the alternative k-NNG approach). Depending on the dataset, and the actual distribution of its data, the weighting required to achieve preferred results may differ. And so it is usually an iterative process to arrive at the final configuration. Also, note that not all undesirable layouts are due to the balance between different data types. Sometimes data may simply be distributed in such a way that no "clean" layout can be calculated that would still reflect the truth about the similarities of dataset rows. In other cases it may require tuning of UMAP's own parameters to achieve best results (see [Creating graphs and layout](/docs/recipes/graphs/create/) as well as Graphext's documention of the steps/methods involved, e.g. [embed\_dataset](/docs/steps/prepare/embed/embed_dataset/)).
# Model training and evaluation
Source: https://docs.graphext.com/concepts/ds-concepts/model-training-evaluation
Amongst existing “machine learning as a service” tools, Graphext aims to be the easiest and most intuitive tools you can use to quickly train a prediction model on tabular data. While simply fitting a ML model to data is relatively straightforward, whether in Graphext or elsewhere, a *good* strategy to create the *best* model, making the most use of your data, should consider two important aspects:
* *Model evaluation:* how to estimate the model’s future performance on unseen data
* *Hyperparameter tuning*: how to best select certain parameters of the model that are not directly learned from data
We here document Graphext’s strategy for model training, tuning and evaluation, which we hope is flexible enough for most use cases, while not overly complex to understand.
For a more in-depth treatment of some of the topics mentioned here see the references in the final section, particularly “Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning" [Raschka (2018)](https://arxiv.org/pdf/1811.12808.pdf) for a more conceptual overview, or [scikit-learn’s overview](https://scikit-learn.org/stable/modules/cross_validation.html) from a coding perspective.
The following sections are organised from most simple to most complete model training strategies. We strive to explain these in a way that is useful to any ML practitioner, while also showing how to configure them in Graphext in particular.
## Introduction
To begin with, let’s clarify the scope of this article. Firstly, we will talk here about supervised ML models mostly; i.e. models which given some samples, each described by a set of features, and corresponding labels, will predict unknown labels for samples it hasn’t seen before. So this could be a model learning from past bank customer’s financial behaviour to predict a person’s credit risk (numerical prediction / regression); or a model predicting whether or not an image contains hot dogs (classification).
Secondly, what we mean by *model evaluation* is estimating how good our model will be at predicting future, unseen data. To do this we need two things:
1. A metric, assigning a numerical score to our model indicating how good its predictions are. Metrics are usually calculated by comparing some samples’ true labels with those predicted by the model. This could be something like *[accuracy](https://en.wikipedia.org/wiki/Accuracy_and_precision#In_classification)* (proportion of correctly predicted labels), or the [mean squared error](https://en.wikipedia.org/wiki/Mean_squared_error). The appropriate metric may depend on the use case (e.g. minimising the rate of false negatives may be more important than false positives in a medical diagnostic test; while the opposite may be true in other scenarios).
2. Some data the model hasn’t seen during its training. If we evaluated the model using data it already “knows”, we may overestimate how good it will perform on truly new data. Nothing would prevent it from simply memorising the data is has been presented, instead of learning to generalise, i.e. to learn the patterns and relationships between features and labels.
How to make best use of our data to both train and evaluate a model, while making sure our estimate of its performance isn’t overly optimistic, is what we refer to as a *training strategy*. Let’s start with the simplest possible strategies and build up towards the more complicated cases step-by-step.
## Simple model training and evaluation
Let’s first consider training a model’s internal parameters only, while leaving its hyperparameters fixed, e.g. using its default values, or selecting them manually based on experience (we’ll talk more about hyperparameters in later sections).
### No evaluation
In principle, we could simply use all our data to train a model, without evaluating its performance. I.e. the simplest possible (but not advisable) training strategy is simply:
![Train without
evaluation]()
Here by “entire dataset” we mean a tabular dataset that contains N samples as rows, M features as columns, and a corresponding set of labels (one per sample). In code, these are often referred to as `X` (NxM samples) and `y` (N labels).
So in the simplest case, we simply pass our model all available samples and labels to learn from. We will have no idea if it’s predictions are any good. Or not yet. The only imaginable use case for this would be if additional data for evaluation would become available later, separately, so that at this point in time all we can do is fit our model blindly.
If we measure the "complexity" of our training strategy by the number of times a model is fit to data, this simplest strategy has a complexity of 1, since we fit the model exactly once using all data.
In Graphext, we can train a model like this by simply leaving its configuration empty, or almost. For example, training an unspecified model without evaluation is simply:
```python theme={null}
train_classification(ds, {"target": "churn"}) -> (ds.pred, "my-model")
```
For detailed documentation, see for example [https://docs.graphext.com/steps/prepare/model/train\_classification/](https://docs.graphext.com/steps/prepare/model/train_classification/)
In Graphext’s training steps we always pass features and labels together as a single dataset (since you’d usually have them together in the same CSV file or database table), and indicate the column containing labels using the `target` parameter.
If no particular model (CatBoost, linear regression etc.) is configured, Graphext will automatically select a default (best) model for the task (classification/regression). The task itself will be determined by the data type of the target column (the labels): classification if the target is categorical (or boolean), and regression if it is numerical.
The above example, e.g., will train a CatBoost classifier to predict the `churn` variable in the dataset `ds`. It will output predictions for the very same samples used to train it (as a new column `pred` in the dataset `ds`, and save the model under the name `“my-model”` for future use.
Since we haven’t asked for model evaluation, and since we have used all data to train our model, the only thing we can measure is how well the model can predict labels for the same samples used to train it. By default Graphext will pick some appropriate metrics for the task and report these as the “train metrics” in the Models section of your project. Note that these are useless as estimates of the model’s real performance. Their only purpose is to gain some insights into whether the model was able to learn anything at all from the data. I.e., if it’s accuracy is bad even on the training set, then either the data doesn’t containing any learnable patterns, or the model is not powerful enough to find them (or to memorise them).
### Holdout method
If all we need is some unseen data to evaluate our model, the simplest possible strategy is to split the dataset into two parts. We use one part to train our model and the other to evaluate it:
![Train without
evaluation]()
Note that when we refer to *splits* of the data, each split is meant to contain both the samples and their features, as well as the corresponding labels. Here, in a first step, the model is fit using samples and labels in the **train** split. The fitted model is then used to make predictions for the samples in the **test** split. These predictions are compared with the test split’s true labels to calculate the estimated generalisation performance of the model.
Now that we have an “unbiased” performance measure for our model (i.e. one calculated using unseen data), we are free to use all available data to create the final model. So in a second step we fit the same model again using the *entire* dataset.
Note that since the final model has been trained with *more* data than was
used for its evaluation, if at all it should be slightly *better* than what we
estimated in the first step. But this is a much better situation than
potentially having a *worse* model, which could happen if we evaluated our
model without first reserving some test data.
In terms of complexity, in this strategy the model needs to be fit twice (once for evaluation and once more for the final model), so we could say its complexity is 2.
In the above diagram we have somewhat arbitrarily selected consecutive samples
at the beginning and end of the dataset for our train and test splits
respectively. By convention, we assume here that the dataset has either been
shuffled already (while preserving the correspondence of samples and labels),
or that it has no intrinsic order. We could also have illustrated the shuffled
and split data like this:
![Shuffle of
dataset]()
But it will be more convenient to assume that data is shuffled already and use consecutive
blocks of data as splits in diagrams from here on.
To configure the simple holdout strategy in Graphext, we can pass the following parameters to one of our model training steps:
```python theme={null}
train_classification(ds, {
"target": "churn"
"validate": {
"n_splits": 1
"test_size": 0.25
}
}) -> (ds.pred, "my-model")
```
This configuration tells us that we want to `validate` the model during training, and that we want to do so by splitting the data once into two parts (`”n_splits”: 1`). It also asks that the test split contain 25% of the samples (`”test_size”: 0.25`), meaning the remaining 75% of samples will be used for training.
In all cases, independent of any configuration, the final model in Graphext will always be trained on *all* data, so step B in the above diagram is always implicit.
### Cross-validation
When we have a lot of data, we may simply reserve a proportion of the data for testing and use the remaining data to fit its parameters, as mentioned above. However, when a dataset is already small, this means fitting the model on an even smaller part of it, which may not be enough given its complexity (usually, the more parameters a model has the more data is needed to optimise it). In addition, evaluating the model on a single random (and small) proportion of the original dataset may result in unreliable estimation (high bias), as it is not guaranteed that the distribution of data in the test part is similar to that in the training part (or to the greater “population” the samples come from).
To remedy this, a common method to *evaluate* a model’s performance on limited data, and the one used by default in Graphext, is to use *cross-validation (CV)*.
**K-fold cross-validation**
Perhaps the most common form of cross-validation is the K-fold CV. The idea here is to split the dataset into K *folds*, and then use K-1 folds for fitting the model and the remaining fold to evaluate the generalisation performance of the model on data it hasn’t seen before.
For example, in a 5-fold cross-validation we divide the dataset into 5 equal-sized, non-overlapping parts, each containing 20% of the samples. We then run 5 iterations and in each:
* select 4 parts of the dataset (80%) to fit the model
* select 1 part of the dataset (20%) to evaluate its performance
![cross validation for
model]()
We then report the average of the model’s performance on the 5 test folds as the expected performance of the model on unseen data. The advantage of this method is that the model is guaranteed to get evaluated on all available samples.
A complete strategy for using k-fold cross-validation to train and evaluate our model then looks like this:
![3 fold CV
strategy]()
We use k-fold cross-validation to estimate the model’s performance, and then fit it again using the whole dataset. The "complexity" of this strategy is thus K + 1, where K is the number of folds in the cross-validation.
To stress a point already made above, to make best use of all the data
available, the final model will always be fitted again on the entire dataset.
This means the estimated performance may be slightly pessimistic, as the model
may not have reached its maximum capacity when fitted with only ⅘ of the
dataset, for example (perhaps with more data the model would do better).
In Graphext, we can configure cross-validation simply by omitting the `test_size` parameter we used in the holdout method:
```python theme={null}
train_classification(ds, {
"target": "churn"
"validate": {
"n_splits": 3
},
"params": {
"C": 1,
"gamma": "scale"
}
}) -> (ds.pred, "my-model")
```
We don’t need the `test_size` parameter in the `validate` section, because k-fold cross-validation splits the datasets into `n_splits` *equal-sized* parts always. Conversely, not providing the `test_size` parameter is how we indicate in Graphext that we want *k-fold* cross-validation, rather than the *shuffle-split* method.
Note we also introduced configuration for selecting some of the model’s hyperparameters by hand. If you don’t want to *tune* them (we will learn how in below sections), you can either leave them at their defaults, or provide constants using the `params` field.
**Repeated holdout cross-validation**
An alternative to *k-fold* cross-validation is to independently split the dataset k times into two random train and test sets. E.g. 5 such *shuffle-split* iterations, maintaining a proportion of 80% training data and 20% test data, may look like this:
![Repeated
Holdout]()
Note that in this case the train/test proportion is independent from the number of splits, i.e. we have the flexibility to e.g. evaluate the model 50 times on random 75%–25% splits of the data. However, it is not guaranteed that the model sees all the data in the process, nor that the splits are different from each other. This method is also sometimes called *shuffle-split* or *Monte Carlo cross-validation*.
Schematically, the complete strategy for using repeated holdout validation to train and evaluate our model then would be:
![Repeated Holdout
Full]()
The complexity of this strategy is also $K + 1$, as all that's changed is how we split the data, not the number of times we do it.
The *holdout* strategy mentioned above can be seen as a special case of the *shuffle-split* with a single iteration only.
![Single shuffled
split]()
Note that this is different from the special case of a *2-fold cross-validation*, which would also split the dataset only once, but into two equal parts containing 50% of the data each. It would then use two iterations to fit the model on one half while evaluating it on the other:
![Single shuffled
split]()
The repeated holdout (shuffle-split) is configured very similar to the k-fold CV in Graphext. We simply specify the desired `test_size` of each iteration.
```python theme={null}
train_classification(ds, {
"target": "churn"
"validate": {
"n_splits": 3
"test_size": 0.2
},
"params": {
"C": 1,
"gamma": "scale"
}
}) -> (ds.pred, "my-model")
```
## Model tuning and evaluation
Many ML models have so-called *hyperparameters* that determine *exactly how* the model learns from data. Basic regression models e.g. fit their coefficients to data such as to minimise a certain loss metric. A *regularised* regression additionally implements a penalty on the coefficients (e.g. to keep the coefficients small, or to use fewer coefficients if possible). The strength of this penalty is one such hyperparameter. The maximum allowed depth of a decision tree, or the number of decision trees in a random forest are other examples.
If you’re lucky, the model in question works well out of the box, with all hyperparameters at their default values. Or if you have a lot of experience training a specific kind of model, you may have some intuition about values that work best in specific scenarios. If neither is the case, or you feel your model could or should perform better than what you’re seeing with the default hyperparameters, you may want to *tune* them. *Tuning* here simply means finding their values automatically, and such that the performance of the model is optimised. In practice, this means using data to select the best from a number of *candidate* *models* having different hyperparameter values.
We may be tempted to simply use the same methodology as explained above to find the best hyperparameters and evaluate our model’s performance. We could e.g. fit 3 different model *candidates* using k-fold cross-validation and select the one that on average had the best performance. The question then arises what its estimated performance would be on *unseen data*. If we simply reported the average from our cross-validation, we would be cheating. The estimate would be biased, because we have used the same data to identify the best model (i.e. to select from our candidates and train it), and to estimate its performance. I.e. we haven’t reserved any data to stand in for future, *unseen* data.
The correct way to both *tune* a model’s hyperparameters *and estimate* its generalisation performance, is to use *nested* cross-validation. The general idea is to evaluate the complete training procedure (hyperparameter selection and model fitting) as we would do in a normal cross-validation, but in each iteration of the evaluation, we split the training set again using an inner cross-validation loop to pick the hyperparameters in a robust way.
But instead of directly jumping in to this rather complex strategy, let's build towards it step-by-step starting from simpler strategies.
### No evaluation (don’t do this) ⛔
As we mentioned above, we don’t recommend fitting or tuning a model without evaluating its generalisation performance. Do this only if you plan to collect more data and evaluate the model later on. Having said that, we *can* tune a model, using the holdout method or cross-validation, and simply report the same performance we used to pick our hyperparameters as a (**bad**) estimate of future performance.
#### Holdout method for tuning without evaluation
The simplest method for tuning our hyperparameters would be to use a single holdout set to pick the best from a set of candidate hyperparameter settings:
![Single shuffled split]()
This is analogous to the simple holdout method mentioned above, but instead of evaluating a *single* model and reporting its performance on the test set, we evaluate *multiple* model candidates (hyperparameter combinations), and use the test set to pick the best among them.
We might perhaps be tempted then to report either the *average* performance on the test set, or the *best* model's performance as our expected generalization ability. But this wouldn't be a good idea. The estimated performance will likely be optimistic, as we used the same data to select between models and to evaluate their generalisation. I.e., we haven’t tested our model training strategy on *unseen* data.
In this strategy our model needs to be fit to data H + 1 times, where H is the number of different hyperparameter combinations to try (H candidates on the training split, and the final model on all data).
To tune hyperparameters in Graphext, simply add a `tune` section to the step’s configuration containing the names and ranges of parameters to explore, like so:
```python theme={null}
train_classification(ds, {
"target": "churn"
"tune": {
"strategy": "grid",
"params": {
"C": [1, 10],
"gamma": ["scale", "auto"]
},
"validate": {
"n_splits": 1,
"test_size": 0.8
}
},
}) -> (ds.pred, "my-model")
```
Note that the `validate` section in this code snippets is located *inside* the `tune` section. This is to indicate that this is the strategy we want to use to select between different hyperparameter settings, not to evaluate generalisation performance. It has the same name `validate`, because it accepts exactly the same parameters (`n_splits`, `test_size` etc.).
The above configuration will split the dataset once into 80% of samples to be used to train our hyperparameter candidates, and 20% to evaluate and pick the winner. Any performance metrics reported back in the Models section will be biased and optimistic, since we haven’t reserved any data for testing.
#### Cross-validation for tuning without evaluation
We can also use cross-validation instead of the holdout method to select the model's hyperparameters without (properly) evaluating its performance:
![Single shuffled split]()
This works the same as the holdout for model tuning, but instead of comparing different hyperparameter combinations on a single train/test split of the data, we compare them using the average over multiple folds of the data. Note, that since we still don't test our procedure on unseen data, the same caveats regarding bias and overly optimistic metrics apply here as well.
In this strategy the model needs to be fit (K \* H) + 1 times in total, for H different candidates and K folds in our cross-validation. A 5-fold cross-validation for exploring 4 different hyperparameter combinations, for example, would result in a total of 21 model fits.
As before, in Graphext we simply omit the `test_size` parameter and select the number of splits (`n_splits`) to be used for cross-validation:
```python theme={null}
train_classification(ds, {
"target": "churn",
"tune": {
"strategy": "grid",
"params": {
"C": [1, 10],
"gamma": ["scale", "auto"]
},
"validate": {
"n_splits": 3,
}
},
}) -> (ds.pred, "my-model")
```
## Three-way holdout
The simplest strategy to tune *and* evaluate a model on unseen data is the *three-way holdout*. This is a simple extension of the holdout method mentioned in the beginning. It splits the dataset once into dedicated *training*, *validation* and *test* sets. This setup is often used in deep learning contexts, where fitting a single model is very expensive but datasets are huge:
![Single shuffled split]()
We evaluate the performance of our model training strategy by:
1. fitting our candidate models on the **training** set
2. picking the best candidate by evaluating them using the **evaluation** set
3. calculating the final score of the winning candidate on the **test** set after having re-fit it on the combined **training** and **evaluation** sets
Having an estimate of our model’s performance, we can then apply the same strategy of picking the best hyperparameters using a simple two-way holdout split (combining the train and eval sets), and finally train the model using the best hyperparameters on the whole dataset.
Another way to describe the same strategy, one more closely matching the implementation and configuration in Graphext, would be to say that we split the dataset once into training and test splits for evaluation, and then train the model (including the tuning of hyperparameters), by splitting the training set again into train and evaluation splits.
In this strategy the model needs to be fit (H + 1) + H + 1 = 2H + 2 times in total, for H different hyperparameter combinations. Selecting between 4 different candidates, for example, would result in a total of 10 model fits.
In Graphext a single random holdout split is configured by setting `"n_splits": 1` and selecting the proportion allocated for testing (`”test_size”: 0.2`, e.g.). Since we want to use a single split to pick our hyperparameters, and a single split again for evaluating, we can combine these in the inner and outer `validate` sections of the configuration:
```python theme={null}
train_classification(ds, {
"target": "churn",
"tune": {
"strategy": "grid",
"params": {
"C": [1, 10],
"gamma": ["scale", "auto"]
},
"validate": {
"n_splits": 1,
"test_size": 0.2
}
},
"validate": {
"n_splits": 1,
"test_size": 0.25
}
}) -> (ds.pred, "my-model")
```
This would reserve 25% of data for testing. The remaining 75% would be split again into 80% for training and 20% for validation (model selection).
## Holdout cross-validation
Slightly more robust than the previous version, this is essentially the *holdout* method for model evaluation (dedicated train and test sets), but using cross-validation for tuning by splitting the training set repeatedly (in the previous method we simply split it once):
![Single shuffled
split]()
Note that by definition we have only used a single split to evaluate our whole training procedure, which in this case includes hyperparameter tuning. This can result in a biased estimate of the model's performance. If the dataset as a whole is larger enough, and with it the holdout set, it may be sufficient. Otherwise we can address this with the nested cross-validation approach explained in next sections.
Using a *K*-fold cross-validation to select between *H* different hyperparameter candidates, in this strategy the model needs to be fit (K \_ H) + 1 times to estimate its performance, another (K \_ H) times to pick the best hyperparameters, and a final time to fit the best model using all data. This makes for a total of 2HK + 2 model fits. With K=5 and H=4, for example, this adds up to 42 fitting iterations.
In Graphext, to use a single shuffle-split partitioning of the dataset for evaluation, select `"n_splits": 1` and a `test_size` parameter in the outer validate section. To use cross-validation in the inner loop for hyperparameter selection, provide only the number of desired folds (`"n_splits": 5` here):
```python theme={null}
train_classification(ds, {
"target": "churn",
"tune": {
"strategy": "grid",
"params": {
"C": [1, 10],
"gamma": ["scale", "auto"]
},
"validate": {
"n_splits": 5
}
},
"validate": {
"n_splits": 1,
"test_size": 0.2
}
}) -> (ds.pred, "my-model")
```
The above would reserve a random 20% of holdout data for testing, and use the remaining 80% for training, where training consists of a 5-fold cross-validation to select the best hyperparameters.
### Nested cross-validation
*Nested* cross-validation addresses the issue of wanting to use cross-validation for both
* reliably picking hyperparameters (instead of relying on a single split to select the winner)
* estimating the expected performance of the final model
without leaking data used in hyperparameter selection into our estimations.
This is somewhat tricky to get right. Perhaps the easiest way to understand nested cross-validation is to treat the tuning of the model’s hyperparameters (candidate selection) as part of the regular training procedure. In essence, we treat our model as a kind of meta-model, which now consists of its normal internal parameters as well as its hyperparameters, and training the model simply means fitting both types of parameters given some data.
Seen this way, *evaluating* the generalisation performance of our meta-model does indeed simply consist of a k-fold cross-validation as explained above. E.g. we split the data into 5 equal parts and then iteratively use 4 parts to identify the best model (hyperparameters), and the fifth part to test its performance:
![Single shuffled
split]()
The average of the 5 folds then is how we expect our combined hyperparameter tuning and model fitting procedure to perform on unseen data. Note that the best model, i.e. the best combination of hyperparameters, may be different in each of the k iterations. But this doesn’t matter. We are *not* selecting any of the “winners” from each iteration as our overall best model. The *only* purpose of the cross-validation is to estimate the *generalisation performance* of our overall training procedure, which now includes the hyperparameter tuning.
#### Inner loop
But, *how exactly* do we select the best model in each iteration of our cross-validation loop? As the name suggests, in *nested cross-validation* we use an *outer loop* to evaluate our overall model training procedure, and a second *inner loop* to select hyperparameters (tuning). I.e. for each *outer* loop iteration, we split the training set again into k parts, use k-1 parts to *fit* our different candidates (hyperparameter combinations), and the kth part to *evaluate* each candidate’s generalisation, like so:
![Single shuffled
split]()
In each *inner* loop, we then select as the winner the model with the best average performance across the evaluation folds. We train this model again using all data in the outer loop’s tuning set, evaluate it on the test fold, and report the average of all winners across the outer loop as our best estimate of the overall training procedure.
#### Candidate selection
We can zoom further in to get a clearer idea of how the best candidate is selected in each outer loop iteration. Here is one such iteration:
![Single shuffled
split]()
If we wanted to tune e.g. 2 hyperparameters of our model, and for each parameter try 2 different values, this would lead to 4 candidate models (4 different combinations of hyperparameters). We may e.g. train a [support vector machine](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC) with regularisation strengths C in `{1,10}` and values of γ in `{scale, auto}`. As shown in the figures above, the winner of each outer loop is the candidate with the best average performance across the inner loop evaluation folds.
#### Final model
As mentioned above, we don’t select any of the winners from our nested cross-validation as our final model. As in the simple case, once we have an estimate of future generalisation performance, we want to make sure we use as much data as is available to fit our final model.
Now, keeping in mind again that training our “meta-model” consists of tuning the model’s hyperparameters using cross-validation (the inner loop basically), our final model is trained exactly like that:
* Use a single k-fold cross-validation over the *whole dataset to* pick the best hyperparameters
* Use these hyperparameters to train a single model on all available data
#### Summary
Schematically, then, the whole procedure of using nested cross-validation for model tuning and evaluation looks like this:
![Single shuffled
split]()
Whenever you don’t have a huge amount of data, and execution time is not a great concern, we recommend to go for the “full monty” and use nested cross-validation for model selection and evaluation. It can be somewhat slow though. If $H$ is the number of different hyperparameter combinations to try,
and $N$, $K$ the number of folds in the outer and inner cross-validation loops, this requires fitting the model:
* $N * K * H$ times to estimate performance
* $K * H$ times to select the best hyperparameters
* 1 time to train the final model
This makes for a total of $NKH + KH + 1$ fitting iterations. For example, a 3x5 nested cross-validation and 4 different candidate models would result in 96 model fits.
From a coding perspective, and taking [scikit-learn](https://scikit-learn.org/stable/modules/cross_validation.html) as an example, our “meta-model” corresponds to simply wrapping our original model in a `GridSearchCV` object (which internally uses cross-validation to find the best hyperparameters among a set of candidates). We then use a simple cross-validation to evaluate the meta-model’s generalisation performance, and refit it to the whole dataset to create the final model (also see [complete example in scikit-learn](https://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html#id2)):
```python theme={null}
inner_cv = KFold(n_splits=3, shuffle=True, random_state=i)
outer_cv = KFold(n_splits=5, shuffle=True, random_state=i)
hyper_params = {"C": [1, 10, 100], "gamma": [0.01, 0.1]}
model = SVC(kernel="rbf")
metamodel = GridSearchCV(estimator=model, param_grid=hyper_params, cv=inner_cv, refit=True)
generalization_score = cross_val_score(metamodel, X, y, cv=outer_cv)
final_model = metamodel.fit(X, y, refit=True)
```
Here, `cross_val_score` estimates generalisation of our entire model training and selection procedure (the grid-search CV). The final `metamodel.fit()` (here GridSearchCV’s `fit()`) then picks the best hyperparameters using the *whole* dataset, and refits this best model again on the whole dataset.
Since we want to use k-fold cross-validation in both the outer loop (evaluation) and the inner loop (hyperparameter selection), we simply select the desired number of splits for both:
```json theme={null}
train_classification(ds, {
"target": "churn",
"tune": {
"strategy": "grid",
"params": {
"C": [1, 10],
"gamma": ["scale", "auto"]
},
"validate": {
"n_splits": 3
}
},
"validate": {
"n_splits": 5
}
}) -> (ds.pred, "my-model")
```
This would split the data 5 times (5-fold cross-validation) in the outer loop to evaluate generalisation performance, and in the inner loop (”tune”) use grid search with 3-fold cross-validation to select hyperparameters.
## Evaluation only
If we already have a trained model we may simply want to evaluate it again on new data; perhaps to test whether it still performs as intended, or whether data drift may have led to a degradation in performance.
This is not possible at the moment in Graphext, but will be in the future using a dedicated step (something like `test_classification` instead of `train_classification` e.g.).
## Summary
We have seen that how to train and evaluate a ML model depends on at least two decisions:
1. whether to include the selection of hyperparameters in the training (tuning)
2. selecting a simple holdout strategy (single split of the dataset intro training and test sets), or a more robust cross-validation (multiple k-fold or shuffle-split iterations).
Both decisions constitute a tradeoff between execution time, resulting model performance and robustness of the *estimated* model performance.
Tuning of hyperparameters should in principle be at least as good, but in most cases hopefully better than not tuning. It may take significantly more time, though, since reliably picking the best hyperparameters depends on fitting the same model to different splits of the dataset many times.
Picking a simple holdout method is faster than cross-validation, since it uses fewer iterations over dataset splits. But for it to be reliable, it is best to use it when the dataset is on the large side. If robustness of the estimated performance is important, and the dataset not large, cross-validation may be more advisable.
There are no hard rules unfortunately, but as a simple heuristic, if execution time is not a great concern, prefer cross-validation over the simpler holdout. Taking inspiration from Sebastian Raschka’s summary in “Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning" [(Raschka, 2018)](https://arxiv.org/pdf/1811.12808.pdf), we can summarize the options available in Graphext in the following figure:
![Single shuffled split]()
As for how to configure Graphext model training, this can be summarised succinctly now:
* To pick a cross-validation strategy (to estimate performance or pick hyperparameters), use
`"validate": {"n_splits": n}`
* To pick a shuffle-split strategy:
`"validate": {"n_splits": n, "test_size": x}`
* The simple holdout method is a special case of shuffle-split using a single split only:
`"validate": {"n_splits": 1, "test_size": x}`
To configure both, the splitting strategy used for evaluation and for tuning, use the same parameters inside and outside the ”tune” section, like so:
```python theme={null}
train_classification(ds, {
"target": "churn",
"tune": {
"strategy": "grid",
"params": {
"C": [1, 10],
"gamma": ["scale", "auto"]
},
"validate": {
"n_splits": 5
}
},
"validate": {
"n_splits": 1,
"test_size": 0.2
}
}) -> (ds.pred, "my-model")
```
This will use 5-fold cross-validation to pick the hyperparameters, and a single holdout split with 20% of the sample for evaluation.
Many classification problems are defined by target variables (labels) with considerable *imbalance*. The different classes of the target variable are represented in significantly different proportions in the dataset. In this case, it is usually a good idea when splitting the dataset to try and maintain the same proportions in each split, a method called [*stratified sampling*](https://scikit-learn.org/stable/modules/cross_validation.html#cross-validation-iterators-with-stratification-based-on-class-labels). Graphext applies stratified k-fold cross-validation or shuffle-split by default when the target variable is categorical (classification).
The intention of this overview has been to introduce the principal ways that Graphext trains and evaluates ML models. We haven’t touched on many finer details regarding the tradeoffs in bias and variance of the mentioned strategies etc. Check the below reference for a deeper scientific understanding of these and related method.
## References
**Conceptual Overview**
* Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning [(Raschka, 2018)](https://arxiv.org/pdf/1811.12808.pdf). Also see [complementary notebook](https://github.com/rasbt/model-eval-article-supplementary/blob/master/code/nested_cv_code.ipynb).
**Academic**
* On Over-fitting in Model Selection and Subsequent Selection Bias in
Performance Evaluation [(Cawley & Talbot, 2009)](https://jmlr.csail.mit.edu/papers/volume11/cawley10a/cawley10a.pdf).
* Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap [(Kim, 2019)](https://www.sciencedirect.com/science/article/abs/pii/S0167947309001601).
**Scikit-learn**
* [Cross-validation overview](https://scikit-learn.org/stable/modules/cross_validation.html)
* [Nested versus non-nested cross-validation](https://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html#id2)
* [Nested cross-validation chapter in MOOC](https://inria.github.io/scikit-learn-mooc/python_scripts/cross_validation_nested.html)
**Stack Exchange**
* [Model selection and cross-validation: The right way](https://stats.stackexchange.com/questions/64991/model-selection-and-cross-validation-the-right-way?rq=1)
* [How to obtain optimal hyperparameters after nested cross validation?](https://stats.stackexchange.com/questions/254612/how-to-obtain-optimal-hyperparameters-after-nested-cross-validation)
* [Nested cross validation for model selection](https://stats.stackexchange.com/questions/65128/nested-cross-validation-for-model-selection)
* [How to build the final model and tune probability threshold after nested cross-validation?](https://stats.stackexchange.com/questions/232897/how-to-build-the-final-model-and-tune-probability-threshold-after-nested-cross-v/233027#233027)
# Regression model
Source: https://docs.graphext.com/concepts/ds-concepts/regression-model
Start exploring your data and discovering insights in under 5 minutes
# UMAP
Source: https://docs.graphext.com/concepts/ds-concepts/umap
Start exploring your data and discovering insights in under 5 minutes
# Cross Filters
Source: https://docs.graphext.com/concepts/graphext-concepts/cross-filters
Quickly filter and select data
Cross filters are arguably one of the most powerful features in Graphext. It's as
simple as it sounds: it filters your data given some criterion. It's the way in which
we define these criterion that is so powerful.
You can **chain** however many filters you want, hence the 'cross filter' naming. This allows
you to hone in to a very thin slice of your data that responds to only the criterion you've
selected. Changing, removing and iterating on this selection is as simple as a couple clicks,
which makes data exploration extremely fast.
To filter a column, you can simply interact with the little chart that's
associated to it.
In the case you reach to a particular selection you may want to preserve, you can do so by
clicking on the little dropdown arrow in the top left corner. This will save the current selection
as a [segment](/concepts/graphext-concepts/segments).
![Saving a selection as a Segment]()
## Absolute and Relative percentages
Upon filtering, all the other variables react to the filter. They show the relative
percentage of entries that fall into their respective categories, effectively showing
you a real-time distribution of the selected data, but in every other column.
To bring it home, let's see this example.
![Cross Filter Price]()
### Example: Host Acceptance Rate
Upon filtering the price, we see the histogram for `host_acceptance_rate` changed.
Now, it shows a percentage y-scale. The gray bars that sit *in the background*
correspond to the percentage of entries that lie in that bin, had we **NOT** filtered the data.
We can see that the last bar, which corresponds to an acceptance rate range from 100
to 110, goes to just over 60%. That means that over 60% of all the hosts have an
acceptance rate of 100 or more.
The blue bar *on top* indicates the percentage of entries that lie in that bin,
**out of the current selection**. Around 55% of the 58 rows we have selected
lie in that acceptance rate range. That's around \~31 rows.
### Example: Is Super Host
Another example is the `is_super_host` variable, just under it. This variable is
either true (t) or false (f) indicating if the host is marked as a Super Host.
In the whole dataset, around 63% of the hosts are not Super Hosts (f). However,
in our particular selection, this is accentuated. Around 70% of the entries **we have
selected** are not Super Hosts. This can be of relevance, depending the questions
we are asking.
The opposite also applies: the percentage of hosts in our selection is
less than the percentage of hosts in the entire dataset.
## Significant Variables
Also, the [significant variables](/concepts/graphext-concepts/significant-variables) kick-in, showing
what other variables may be interesting in regards to the selected one.
# Insights
Source: https://docs.graphext.com/concepts/graphext-concepts/insights
Save and recall findings in your research
Insights allow you to take a snapshot of your project as some relevant
finding or relation you found. They are stored under the Insights tab
and also allow you to restore the state of the project at that point,
enabling easy replication of all the [cross filters](/concepts/graphext-concepts/cross-filters) and
plot compositions you created at that point.
![Insights in Graphext]()
Pressing the play button in the lower left corner of each insight will
replicate the state of the app at that point, to further investigate
in that line of thought.
Alternatively, insights also serve as, well... insights! Pieces of important
information you found in your data, like relationships between variables or plots.
![Insight of a plot in Graphext]()
Saving an insight of a plot will also save all the customizations made to it,
allowing for quick generation of ideas that can later be properly exported for
sharing or publishing.
# Recipe
Source: https://docs.graphext.com/concepts/graphext-concepts/recipe
The recipe is the backbone of any Graphext proyect
The recipe is a number of [steps](/concepts/graphext-concepts/steps), which
are functions that accept some data and output new, transformed
or enriched data.
Writing a recipe is one way to instruct Graphext on how to build a
project/visualization given some tabular data (others being the use of
the [Wizard](/concepts/graphext-concepts/wizard), or the application of pre-defined recipes).
A recipe can have an arbitrary number of such steps.
When you open the **Recipe Editor**, you will have done so having selected
first a dataset that serves as the main input for the recipe. This dataset
is made available by default with the name `ds`, and so the simplest possible
recipe looks like
```erlang theme={null}
create_project(ds)
```
i.e. a recipe with a single step called `create_project` which accepts a dataset
as input and has no output. This is a special case. Since the result of this step
is the **creation of a project**, it doesn't generate any output that can be further
processed inside the recipe.
However, in practice you'll almost always want to somehow transform or enrich your dataset
and so you'll want to add one or more of the many steps available
in Graphext before the final step of project creation. A complete reference
for all the steps available and how to use them lives under the [API Docs](/api-docs/steps)
# Segments
Source: https://docs.graphext.com/concepts/graphext-concepts/segments
Group selections for easy access
## What is a Segment?
Segments can be thought of as a group of [cross filters](/concepts/graphext-concepts/cross-filters). These allow
us to save an arbitrary amount of filters on several columns that may
make semantical sense.
For example, you could save demographic data in a more approachable way. In the
titanic dataset, we have age and gender in two different variables. Assuming we wanted
to have a coarse perspective on these two factors, like Young vs Old people, and
Men vs Women, we could create a "Demographics" segment.
![Demographics segment in Graphext]()
By creating these four segments, we now have a very quick way to
reach for the "Young Men" category, which would involve selecting
all Male passengers under the age of 25.
This is a simple example, but we could compose an arbitrarily complex
filter, which would make reaching for these specific rows much easier.
## How it works
As we can see, this process just involves creating a new multivalued column
that assigns the name of the segment to the selected rows. If the row was present
in the initial filter, the row gets that category in the column. We can see that some
rows may be included in several filters, hence the multivalued column that can store
an array of different values.
![]()
# Significant variables
Source: https://docs.graphext.com/concepts/graphext-concepts/significant-variables
A quick peek into potential correlations
Significant variables appear when you [filter your data](/concepts/graphext-concepts/cross-filters). By selecting a date range, a
category or a number range in any of the columns, you are selecting a subset of all
the possible data points you have. Depending on what variable you used, this filter can
be related to some other variable. Those variables that show a strong correlation with
the selected one are **significant variables**.
These appear on the top left side of the interface upon filtering.
You can see in here that when filtering the `host_since` column, which indicates
when this Airbnb host first logged in, different correlations appear.
![]()
If we filter quite back in time, `host_is_superhost` correlates more strongly. This
means that hosts need quite a bit of time before actually becoming superhosts.
![]()
On the other hand, when filtering a bit closer to the end of the range, `first_review`
correlates more strongly instead. Which makes sense, since having a *first review* is the
most common event among new hosts.
We are skipping the two first entries on purpose. Correlations on an ID column (`host_id`)
are generally not useful. And a strong correlation against the same variable is also to be expected.
Graphext evaluates all columns anyways!
## How is the score calculated?
![variable relevance score]()
When hovering over the five little bars next to each variable, a score appears. This score tells us how significant the
distribution of this variable is to the distribution of the current selection we've made.
### Manually computing an example
#### Context
Assume we are on a dataset where, among other things, we have a column `month` which has the numbers 1 – 12 for each month,
and a column `season` which has the values "Summer", "Autumn", "Winter", "Spring".
We all know that Spring happens in March, April and May, approximately. If we select the category "Spring", we can see
that the months 3, 4 and 5 show up these blue spikes, whereas the rest of the months do not.
![Selection of spring]()
If we hover over one of these blue bars in this case, this tooltip informs us of several important figures:
![Selection of spring]()
* 3.19K represents the number of rows that have this particular value from this particular column (in this case, the value 5 for column `month`). Or, basically, the grey-ish bar behind the blue one.
* This 3.19K amounts to an \~8.5% of the whole dataset, which has \~37.3K rows.
* Importantly, these 3.19K rows correspond to \~33% of all the 9442 rows we selected when clicking on "Spring". This makes sense: spring spans across 3 months, so May alone has about a third of all the days that comprise the whole Spring.
If we hover over month 6 instead, this shows up:
![Selection of spring]()
There are \~3.07K rows occurring in June, but 0 that have both the value 6 in `month` AND the value "Spring" in `season`.
#### Computing
The way we calculate how different the distributions between month and season are would involve going over each
of these tooltips, subtracting the two percentage values, getting the absolute value, adding them all up
and finally dividing by two.
The full operation would look like this.
1. We add up all the absolute value of the differences:
$$
\begin{align}
&(8.3 - 0) + (8.3 - 0) + (33.3 - 8.3) \\ &+ (33.3 - 8.3) + (33.3 - 8.3) + (8.3 - 0) \\ &+ (8.3 - 0) + (8.3 - 0) + (8.3 - 0) \\ &+ (8.3 - 0) + (8.3 - 0) + (8.3 - 0) \\ &= 149,7
\end{align}
$$
2. Finally, we divide the whole result by 2:
$$
149.7 / 2 = 74.85 \approx 75
$$
if we hover over `month` in the significant variables section:
![Selection of spring]()
We are not justifying the math behind this, since this flies a bit out of the
scope for this short explainer. Check the
[references](/concepts/graphext-concepts/significant-variables#references) if
you really want to dig into the topic.
## Summing up
This computed the distance of the distributions created by your selection and the distributions that are already present
in your data. Each of the different values a variable may have is a distribution on itself that changes when you select
particular rows within your data. Studying these distributions and their differences help us understand our data in better
ways.
### References
The actual calculation that's going on corresponds to the [total variation distance of probability measures](https://en.m.wikipedia.org/wiki/Total_variation_distance_of_probability_measures), for those
interested in the math behind it.
This article: [Is there a difference?](https://www.data8.org/fa15/text/3_inference.html#Total-Variation-Distance) goes into great depth with a practical example calculating and interpreting the result.
# Steps
Source: https://docs.graphext.com/concepts/graphext-concepts/steps
Steps are functions used in the context of the recipe
Steps are functions that tell Graphext how to process your data, in
a pipeline sort of way. They live in the [recipe](/concepts/graphext-concepts/recipe),
as a sorted list of processes to perform on your data. Usually,
the output from one will go into the next, although this is not strictly
necessary.
Some steps output a new column, some just help you set metadata about a column,
some help you export your data, train a model and much more.
Steps are written in a *low code* language specific to Graphext recipes, that resembles
a bit of a mix between Javascript and Python. The recipe editor will help with auto-completion
of the steps, as well as auto-filling their inputs.
Here's an example of a recipe with only two steps:
```erlang theme={null}
extract_json_values(ds.products, {
"path": "name",
"type": "category"
}) -> (ds.productName)
create_project(ds)
```
We can see the `extract_json_values` and the `create_project` steps. The `create_project` step is a special one whose only purpose
is to instantiate the dataset (ds) and make it available for other steps to process it.
We can see the step `extract_json_values` takes two inputs: `ds.products` and
a dictionary-like object as options. `ds.products` is a column on `ds`, made available
by `create_project`. This step will then create a new column on `ds` called `productName`, with the
results of the transformation in it.
This is a very powerful and relatively easy way of processing your data and having it
readily available. Sometimes, this can be quite a bit faster than booting up a classic
python/R notebook, while still providing with much the same functionality.
The complete list for all the possible steps lives in the [API Docs](/api-docs/steps), also at the top of this page.
Do not hesitate to [reach out](mailto:support@graphext.com) if you need any help!
# Tags
Source: https://docs.graphext.com/concepts/graphext-concepts/tags
Make sense of your variables
Tags provide a way to group variables based on their meaning. These groups can then
be used across the application for a more convenient experience.
To learn more about how to deal with tags, check out [how to group variables](/documentation/data-preparation/variable-management-ui-config/group-variables).
## Using tags
Tags can be thought like a **variable group**. For example, in this Airbnb dataset, we have 82 variables, 18 of which
are exclusively related to the host data, such as their ratings, superhost status,
location, response rate and so on. By tagging all these as '**host**', we can quickly access them
from this dropdown in the variable manager, or the main interface.
We also created another tag, called '**location**', for all variables related to coordinate
and neighbourhood data.
![tags main]()
![tags main]()
# Wizard
Source: https://docs.graphext.com/concepts/graphext-concepts/wizard
Transform and enrich your data easily and conveniently
The Wizard is your personal assistant for advanced steps composition when
you need to do some transformation or processing on your data.
The Wizard really just helps you compose [steps](/concepts/graphext-concepts/steps)
within the [recipe](/concepts/graphext-concepts/recipe), but does so in a more
intuitive and opinionated way, when you are not entirely sure of
what to write.
![Wizard]()
The menu guides you through all the steps necessary to accomplish what
you have in mind.
Here you can see how easy it is to make a model that predicts `price` based on
some coordinates and other potentially important factors:
![Model creation through Graphext's wizard]()
And this is all the code it generated for us. Among other things, it mainly extracts some
time components from date-based columns, creates tags, uses the [train\_classification](/api-docs/analyse/train_and_predict/train_classification) and [test\_classification](/api-docs/analyse/train_and_predict/test_classification) steps to train a model with the default configuration and creates a bunch of columns that hold this prediction data.
Some of the intermediate columns are hidden, since they are important for the model but not revelant for human interpretation. Nevertheless, these can be brought back if needed.
```erlang theme={null}
extract_date_component(ds.last_scraped, {
"component": "hour"
}) => (ds.hour)
extract_date_component(ds.last_scraped, {
"component": "month_name"
}) => (ds.month_name)
extract_date_component(ds.last_scraped, {
"component": "weekday_name"
}) => (ds.weekday_name)
configure_tagged_columns(ds[["neighbourhood_cleansed", "host_verifications", "host_total_listings_count", "longitude", "latitude", "price"]],
{
"Target": [
"price"
],
"Factors": [
"latitude",
"longitude",
"host_total_listings_count",
"host_verifications",
"neighbourhood_cleansed"
]
})
train_classification(ds[["price", "neighbourhood_cleansed", "host_verifications", "host_total_listings_count", "longitude", "latitude"]],
{
"target": "price",
"model": "CatboostClassifier",
"params": {
"depth": 6,
"nan_mode": "Min",
"iterations": 1000,
"one_hot_max_size": 10,
"max_ctr_complexity": 2,
"boosting_type": "Plain"
},
"validate": {
"n_splits": 5,
"metrics": [
"accuracy"
]
}
}) => (ds.gx_prediction,
"ds-model-yFqI")
test_classification(ds[["price", "neighbourhood_cleansed", "host_verifications", "host_total_listings_count", "longitude", "latitude"]],
"ds-model-yFqI",
{
"refit": true,
"split": {
"test_size": 0.2
},
"target": "price"
}) => (ds.gx_prediction,
ds.prob,
ds.Error,
ds.split)
configure_column_metadata(ds.Error,
{
"label": "Error",
"description": "Whether the predicted class was correct or wrong"
})
configure_column_metadata(ds.gx_prediction, {
"label": "Prediction",
"description": "Prediction made for the target varible"
})
configure_tagged_columns(ds[["split", "prob", "Error", "gx_prediction"]],
{
"Output Variables": [
"gx_prediction",
"Error",
"prob",
"split"
]
})
configure_columns_order(ds.price,
ds.gx_prediction,
ds.Error,
ds.prob,
ds.split,
ds.latitude,
ds.longitude,
ds.host_total_listings_count,
ds.host_verifications,
ds.neighbourhood_cleansed)
configure_column_visibility(ds.Error, {
"graph": "pinned"
})
configure_column_visibility(ds.prob, {
"graph": "pinned"
})
configure_column_visibility(ds.split, {
"graph": "pinned"
})
```
# Advanced Filter Queries
Source: https://docs.graphext.com/documentation/data-exploration/advanced-filter-queries
Surgically precise selection and filtering
## Custom Query Selection
Cross filters also allow you to make arbitrary selections, with a specific, but
simple syntax.
![query string cross
filter]()
These are the operators you can use:
* logical: `AND, OR, NOT`
* numerical: `<, >, <=, >=`
* text: `REGEX, SUBSTR, FUZZY`
* frequency selection: `TOP, FREQ`
* sorting methods (to be used with `TOP`): `BACKGROUND, FOREGROUND, UPLIFT, TFIDF, ORDINAL`
* null values: `NULL`
* statistics: `MIN, MAX, MEAN, P25, MEDIAN, P75`
The query you build and the syntax available will depend on the type of the variable you are filtering:
* **All** column types accept the **logical** operators to build complex queries using other operators, as well as the `NULL` operator, which exclusively returns null rows.
### Logical operators
Logical operators can concatenate simpler expressions to build more complex
ones.
This selects all rows that have a `age` value greater than 10 and less than 55:
```erlang theme={null}
>= 10 AND <= 55
```
we could be interested in two disjoint age brackets, like so:
```erlang theme={null}
( >= 10 AND <= 55 ) OR ( >= 80 AND <= 90 )
```
this will return all rows that are greater than 10 and less than 55 as well as all rows
that are greater than 80 and less than 90, effectively returning two disjoint
age brackets at the same time.
Or we could remove a specific interval within the age bracket:
```erlang theme={null}
( >= 10 AND <= 55 ) AND NOT( >= 30 AND <= 40 )
```
effectively selecting all rows between 10 and 55 but removing all rows between
30 and 40, leaving a "gap" in between.
![selection gap]()
Logical operators work with any kind of variable, so you can do this too:
```erlang theme={null}
(FUZZY('SLEEP') OR FUZZY('TREATMENT'))
AND
FUZZY('MASK')
```
selecting all rows that contain either "sleep" or "treatment" and also contain the word "mask".
![fuzzy]()
You can focus on null values:
```erlang theme={null}
NULL
```
or exclude them!
```erlang theme={null}
NOT(NULL)
```
### Categorical & Text
In categorical or text columns you can search for specific words (tokens) or categories. You can combine searches using boolean logic (i.e. string together multiple conditions using the AND/OR keywords).
**Categorical** and **Text** variables accept the `TOP`, `FREQ`, `FUZZY`, `SUBSTR` and `REGEX` operators. They work similarly since
both are based on text: categories are just very short expressions, whereas text
tends to present a longer format.
* On any text-based variable, you can simply ask for `ball`, which will return all rows that contain the word `ball` by itself.
* `FUZZY(ball)` will return all rows that contain the word `ball`, whether `ball` is part of other words or appears by itself.
* `FUZZY` is case insensitive and [normalizes all input (a.k.a ASCII folding)](https://unicode-org.github.io/icu/userguide/transforms/normalization/) before searching.
* `SUBSTR(ball)` will return only rows that contain `ball` as part of a word, but not by itself.
* `REGEX()` will accept a [regular expression](https://en.wikipedia.org/wiki/Regular_expression) string to match more complex patterns.
![fuzzy ball]()
![ball]()
* `FREQ` selects those terms whose frequency is greater or equal than `n`: `FREQ(10000)` selects those values whose frequency is greater or equal to 10K.
* `TOP` selects the top `n` terms in terms of frequency: `TOP(10)` selects the top 10.
* we can, however, modify `TOP`'s behavior by saying `TOP(10, FOREGROUND)`, which would select the top 10 **out of the current selection we have made**.
* `TOP(10, UPLIFT)` selects the top 10 after sorting them by how different the frequency is between the selection and the whole dataset. These operators are the same as the ones mentioned in the [sorting](#sorting) section.
Keep in mind that the sorting methods will not visually sort the elements in
the cross filter, but they will just return the relevant elements once they
are filtered and sorted. To sort the elements visually, you can head to the
[sorting](#sorting) section.
![frequency]()
Select rows where the `department` column exactly contains "engineering":
```erlang theme={null}
engineering
```
on the other hand, rows that include "engineering", maybe among other terms:
```erlang theme={null}
FUZZY(engineering)
```
or where the `text` column exactly matches "he" and "she":
```erlang theme={null}
he AND she
```
or filter the top four most frequent categories in the `department` column:
```erlang theme={null}
TOP(4)
```
select all rows that have over 20K occurrencies:
```erlang theme={null}
FREQ(20000)
```
### Numerical & Dates
Numerical and Date columns behave similarly, since, internally, they are just
storing numbers. This section explains valid syntax and examples for each type.
#### Numerical
**Numerical** variables will naturally accept the numerical operators, as well as the statistical operators.
![numerical select]()
Generally, the statistical operators provide a quick shortcut to the most relevant sections of a distribution:
* MIN: selects all rows that have the same value as the minimum value found
* MAX: selects all rows that have the same value as the maximum value found
* MEDIAN: selects all rows that have the same value as the median
* P25: selects all rows that have the same value as the first quartile
* P75: selects all rows that have the same value as the third quartile
To build a query spanning from the first quartile to the third, you can simply say `>= P25 AND <= P75`, which returns all rows that are both
greater or equal than the P25 and less or equal than the P75. This effectively returns the [the interquartile range](https://en.wikipedia.org/wiki/Interquartile_range).
In numeric columns, numbers can also be specified using scientific notation. The following two numbers are both valid and represent the same number: **145000** and **1.45e5**.
Queries in numeric columns support **=**, **>**, **>=**, **\<** and operators.
To select rows whose age field falls into the range 10 to 55:
```erlang theme={null}
>= 10 AND <= 55
```
To match a specific number:
```erlang theme={null}
12
```
The = operator is implicit in the above query, so the following query produces the same result:
```erlang theme={null}
=12
```
Selecting everything BUT a specific number:
```erlang theme={null}
NOT(12)
```
Include ages from 10 up to but not including 55 (exclusive smaller/greater than):
```erlang theme={null}
>= 10 AND < 55
```
Using scientific notation:
```erlang theme={null}
>= 5.6e-4 AND <= 9.35e-2
```
Using a computed statistic, anything over the 25th percentile:
```erlang theme={null}
>= P25
```
Or anything under the 75th percentile:
```erlang theme={null}
<= P75
```
Retrieving the interquartile range:
```erlang theme={null}
>= P25 AND <= P75
```
Combining both constants and computed statistics:
```erlang theme={null}
>= 5 AND < MEAN
```
#### Dates
**Dates** behave in much the same way as numbers. They are specified in the [ISO format 8061](https://en.wikipedia.org/wiki/ISO_8601) standard, where the date part is required and the time part optional. I.e. both the following dates are valid: **2020-02-23** and **2020-02-23T14:30:00**. In the first example, only year, month and date are specified, while the second includes the time also (hours, minutes and seconds)
We can say things like `>= 2018-06-05T11:33:48.554Z AND <= 2021-06-26T05:56:18.172Z`, which means anything *after* June 5th, 2018 at 11:33:48 **and** *before* June 26th, 2021 at 05:56:18, effectively
returning dates within that time interval. Same as with numbers, simply creating a range in the cross filter will generate this query for you to adjust, in case more precision was needed.
Here are some examples of valid date notation:
To select all rows whose "date" field falls into the year 2019:
```erlang theme={null}
>= 2019-01-01 AND <= 2019-12-31
```
Or to match a specific
```erlang theme={null}
2019-01-01
```
The = operator is implicit in the above query, so the following query produces the same result:
```erlang theme={null}
=2019-01-01
```
Select dates before 2020, January 1st, 12:35 PM:
```erlang theme={null}
<= 2020-01-01T12:35:00
```
Dates before the median date in the same column:
```erlang theme={null}
>= 2020-02-23 AND <= MEDIAN
```
And you may of course also combine constants with calculated statistics:
```erlang theme={null}
>= 2020-02-23 AND <= P75
```
Some steps allow the use of *advanced queries* to filter rows in a dataset.
These are the steps that currently support it:
* [create\_filter\_insight](https://docs.graphext.com/steps/report/create_insight/create_filter_insight/)
* [create\_graph\_insight](https://docs.graphext.com/steps/report/create_insight/create_graph_insight/)
* [create\_plot\_insight](https://docs.graphext.com/steps/report/create_insight/create_plot_insight/)
* [filter\_rows](https://docs.graphext.com/steps/prepare/filter/filter_rows/)
* [segment\_rows](https://docs.graphext.com/steps/prepare/transform/any/segment_rows/)
* [cluster\_network](https://docs.graphext.com/steps/analyse/graph_and_map/cluster/cluster_network/)
* [cluster\_subnetwork](https://docs.graphext.com/steps/analyse/graph_and_map/cluster/cluster_subnetwork/)
# The Compare tab
Source: https://docs.graphext.com/documentation/data-exploration/compare
Explore your data in a massively parallel way
The Compare section offers you a fine grained view of your variables, particularly focusing in how each one
of their values work in relation to other variable's values.
Essentially, you can compare how all the possible values from a column relate to any other column's values, allowing you
to spot potentially interesting relationships.
## A practical example
We can analyze this example to get a sense of how it works.
![the compare tab]()
This dataset holds data on \~20K Airbnb listings in Madrid. We have information about the listing's location, price,
neighbourhood, reviews, information related to the owner and more.
We can see the variable Neighbourhood selected, and two of its values compared: *Casa de Campo* and *Sol*. This allows us
to **compare** the two neighbourhoods very quickly.
You can see how the different aspects of the review process differ between listings in one neighbourhood and the other.
Or, more interestingly, how the price between the two differs, in the leftmost chart:
![the compare tab]()
You can hover your mouse over any point in any chart to reveal more information about it:
![compare hover]()
![compare-hover-2]()
We can see that 41% of all the listings between 50 to 100€ per night are located in Casa de Campo, whereas Sol
holds only \~29% of those in the same price range, making it clear that Sol is a quite more expensive neighbourhood.
# The Correlations tab
Source: https://docs.graphext.com/documentation/data-exploration/correlations
Find out not only what, but why something happens
The Correlations tab offers a bird's eye view of all your columns, creating visualizations
that allow you to spot potential combinations of variables and values that may be of interest.
![correlation tip]()
Upon choosing a variable to analyze, all the other variables are laid out in a grid, with [heatmaps](/documentation/data-visualization/types-of-chart/heatmap) that
inform us of where the distribution of values lies, as well as [boxplots](/documentation/data-visualization/types-of-chart/box-plot) that show us the distribution
under each value of the selected variable.
If you don't see the Correlations tab, you may need to access it from the menu, and, optionally, pin it so
it's always visible!
![correlation tip]()
# Advanced data selection
Source: https://docs.graphext.com/documentation/data-exploration/cross-filters/advanced-selection
Rediscover data exploration interactively
***
## Exploring with cross filters
[Cross filters](/concepts/graphext-concepts/cross-filters) are one of the most powerful tools Graphext offers. They are natural to use, as they
show the distribution of your variables, but enable exploration on different combinations of values,
making the whole interface reactive.
For example, in this dataset holding transactions from an e-commerce, we can filter those transactions made
between 2020 and 2022:
![Cross filter date]()
which leaves us with 72% of the data, 1.1M rows out of 1.6M we have in total.
Notice the relative scale on the right, spanning from 0 to 12% (really it's more like \~13%). That is now telling us how
much of our data lies on each of the bars (called bins).
This (or any) selection affects every other cross filter. This is what makes them so powerful: they all behave like
one single system informing of the different distributions of your variables.
![select gift cards]()
It is worth noting that using cross filters affects the whole state of the application, meaning that Graph and Plot also react
to whatever you are selecting.
## Sorting and filtering
Cross filters can also be sorted and searched, making surgically precise questions a breeze to answer.
### Sorting
You can sort categorical and text variables, in several ways. The default is "by everything", which just means the frequency
of each value sorted in descending order; the most common items appear first.
You also have these other methods available:
* **Selection**: the same as "by everything" but just taking into account the current active selection
* **Uplift**: the difference in frequency between the selection and the whole dataset. Bigger differences will appear first.
* [TF-IDF](https://en.wikipedia.org/wiki/Tf%E2%80%93idf): measures the importance of a term (or category) with respect to the whole dataset.
* **Ordinal**: if you have [provided ordinal information](/documentation/data-preparation/variable-management-ui-config/specify-order-in-column) to your variable, you can sort it this way.
* **Alphabetically**: sort the categories alphabetically in descending order.
![sorting methods]()
For example, say we select this specific demographic of women between the age 18 and 24.
![select young women]()
we can see how the category column changes based on this information, and sorting it according to the most
relevant data. That is, the one that differs most with respect the whole dataset, without selection.
If we sort Category based on Uplift, we see interesting stuff:
![uplift popup]()
![uplift sorted]()
corsets, jewlery and hair extensions come as one of the most distictive results for this specific
subset of data. Which, indeed makes sense.
Remember we are not sorting by frequency (since that's the default), but
rather by how different this distribution is with respect to the original
dataset, with no filters.
These results must be taken with a grain of salt, since most of these bars are representing tens or hundreds of datapoints,
which are completely dwarfed by the scale of the million datapoints we have. While promising, they represent a **very** small portion of our
population. Take this into account in your own research.
### Selecting
Clicking the little magnifying glass in a cross filter will allow you to search through the different values it holds:
![custom search]()
This popup allows you to select any segment belonging to that column. You can
select different rules for searching, like exact match, or contains. This just
translates your choices to an [advanced filter query](/documentation/data-exploration/advanced-filter-queries).
This magnifying glass is only available in text-based variables, like `text` or
`category`. In `numerical` or `date` variables, you can access it via the options
menu → Custom query selection.
![custom search numeric]()
## Re-ordering cross filters
Just in case you missed it, you can [group](/documentation/data-preparation/variable-management-ui-config/group-variables), [pin](/documentation/data-preparation/variable-management-ui-config/pin-variables-menu) and [rearrange variables](/documentation/data-preparation/variable-management-ui-config/arrange-variables-menu), so the most important information
is always where you want it to be.
# Segments
Source: https://docs.graphext.com/documentation/data-exploration/cross-filters/creating-segments
Group selections for easy access
## What is a Segment?
Segments can be thought of as a group of [cross filters](/concepts/graphext-concepts/cross-filters). These allow
us to save an arbitrary amount of filters on several columns that may
make semantical sense.
![Demographics segment in Graphext]()
## Creating a Segment
Creating a segment involves making a selection of an arbitrary number of variables, and then "saving" that selection.
### Step by step
First, make any selection you are interested in, and then head to the Save selection menu, just under the big figure
showing the currently selected rows, in the top left corner of the screen.
![creating segment]()
If you already created a segment, it will appear in the menu. Choose it to save the current selection to the
existing segment. Otherwhise, create a new segment clicking the "New Segmentation" button.
Here in the video we save 3 groups of gender-age demographics for easier access: men in the age 0–25 age range, 25–50 and 50–90.
These are saved in new, specific categories inside a new variable that can be used to select that data in one single click,
or even train a machine learning model.
## Practical example
For example, you could save demographic data in a more approachable way. In the
titanic dataset, we have age and gender in two different variables. Assuming we wanted
to have a coarse perspective on these two factors, like Young vs Old people, and
Men vs Women, we could create a "Demographics" segment.
![Demographics segment in Graphext]()
By creating these four segments, we now have a very quick way to
reach for the "Young Men" category, which would involve selecting
all Male passengers under the age of 25.
This is a simple example, but we could compose an arbitrarily complex
filter, which would make reaching for these specific rows much easier.
## How it works
As we can see, this process just involves creating a new multivalued column
that assigns the name of the segment to the selected rows. If the row was present
in the initial filter, the row gets that category in the column.
Some rows may end up in several filters simultaneously, hence the multivalued column that can store
an array many different values.
![]()
# Overview
Source: https://docs.graphext.com/documentation/data-exploration/cross-filters/overview
Introducing Cross Filters
Cross Filters are powerful interactive representations of your data, present in both the left and sidebars
in the Graphext interface.
Check out this video, where you can learn more about the concept of Cross Filters in 6 minutes:
Here, you can read more about them, what they can do, and what effects they have on the whole experience of
exploring and analyzing data.
Rediscover data exploration interactively
A quick peek into potential correlations
Group selections for easy access
# Significant Variables
Source: https://docs.graphext.com/documentation/data-exploration/cross-filters/significant-variables
A quick peek into potential correlations
Significant variables appear when you [filter your data](/concepts/graphext-concepts/cross-filters). By selecting a date range, a
category or a number range in any of the columns, you are selecting a subset of all
the possible data points you have. Depending on what variable you used, this filter can
be related to some other variable. Those variables that show a strong correlation with
the selected one are **significant variables**.
These appear on the top left side of the interface upon filtering.
![significant variables
UI]()
You'll see 5 groups of tiny rectangles next to a number. These rectangles display a score of how
significant the selection you've made is with respect to how this variable changed. Those variables that
suffer a greater change in their distribution after selecting will have a higher score, meaning they could be quite
relevant to whatever selection you've made.
![significant variables
tooltip]()
## How is the score calculated?
![variable relevance score]()
When hovering over the five little bars next to each variable, a score appears. This score tells us how significant the
distribution of this variable is to the distribution of the current selection we've made.
### Manually computing an example
#### Context
Assume we are on a dataset where, among other things, we have a column `month` which has the numbers 1 – 12 for each month,
and a column `season` which has the values "Summer", "Autumn", "Winter", "Spring".
We all know that in the northern hemisphere, Spring happens in March, April and May, approximately. If we select the category "Spring", we can see
that the months 3, 4 and 5 show up these blue spikes, whereas the rest of the months do not.
![Selection of spring]()
If we hover over one of these blue bars in this case, this tooltip informs us of several important figures:
![Selection of spring]()
* 3.19K represents the number of rows that have this particular value from this particular column (in this case, the value 5 for column `month`). Or, basically, the grey-ish bar behind the blue one.
* This 3.19K amounts to an \~8.5% of the whole dataset, which has \~37.3K rows.
* Importantly, these 3.19K rows correspond to \~33% of all the 9442 rows we selected when clicking on "Spring". This makes sense: spring spans across 3 months, so May alone has about a third of all the days that comprise the whole Spring.
If we hover over month 6 instead, this shows up:
![Selection of spring]()
There are \~3.07K rows occurring in June, but 0 that have both the value 6 in `month` AND the value "Spring" in `season`.
#### Computing
The way we calculate how different the distributions between month and season are would involve going over each
of these tooltips, subtracting the two percentage values, getting the absolute value, adding them all up
and finally dividing by two.
The full operation would look like this.
1. We add up all the absolute value of the differences:
$$
\begin{align}
&(8.3 - 0) + (8.3 - 0) + (33.3 - 8.3) \\ &+ (33.3 - 8.3) + (33.3 - 8.3) + (8.3 - 0) \\ &+ (8.3 - 0) + (8.3 - 0) + (8.3 - 0) \\ &+ (8.3 - 0) + (8.3 - 0) + (8.3 - 0) \\ &= 149,7
\end{align}
$$
2. Finally, we divide the whole result by 2:
$$
149.7 / 2 = 74.85 \approx 75
$$
if we hover over `month` in the significant variables section:
![Selection of spring]()
We are not proving and demonstrating all the math behind this, since this
flies a bit out of the scope for this short explainer. Check the
[references](/concepts/graphext-concepts/significant-variables#references) if
you really want to dig into the topic.
## Summing up
This computed the distance of the distributions created by your selection and the distributions that are already present
in your data. Each of the different values a variable may have is a distribution on itself that changes when you select
particular rows within your data. Studying these distributions and their differences help us understand our data in better
ways.
### References
The actual calculation that's going on corresponds to the [total variation distance of probability measures](https://en.m.wikipedia.org/wiki/Total_variation_distance_of_probability_measures), for those
interested in the math behind it.
This article: [Is there a difference?](https://www.data8.org/fa15/text/3_inference.html#Total-Variation-Distance) goes into great depth with a practical example calculating and interpreting the result.
# AI for data preparation
Source: https://docs.graphext.com/documentation/data-preparation/ai-data-preparation
Start exploring your data and discovering insights in under 5 minutes
# Data cleaning & transformation
Source: https://docs.graphext.com/documentation/data-preparation/data-cleaning-transformation
Start exploring your data and discovering insights in under 5 minutes
# Fill Missing Values
Source: https://docs.graphext.com/documentation/data-preparation/data-enrichments/fill-missing-values
Use a function to fill missing values
This function allows to fill with some value those rows in which there was no data to begin with.
Check out the [replace\_missing](/api-docs/prepare/transform/replace_missing)
step for more information.
## Parameters
* **Column** to fill values in
* **Method** to use
* **Constant value**: use a constant value throughout the whole column
* **Least frequent value**: fill with the least frequent value of the column
* **Most frequent value**: fill with the most frequent value of the column
* **First value in alphabetical order**: fill with the first value of the column in alphabetical order
* **Last value in alphabetical order**: fill with the last value of the column in alphabetical order
* **Next valid observation**: fill with the same value that the next non-null row has
* **Previous valid observation**: fill with the same value that the previous non-null row has
# Group Similar Semantics
Source: https://docs.graphext.com/documentation/data-preparation/data-enrichments/group-similar-semantics
Group together words with similar meanings
This function allows us to merge together words that mean the same. We are
talking cases like "trainers" and "sneakers", but many other cases may apply.
This helps boiling down the main message to its core, making it clearer and
more useful for models and/or other purposes.
Check out the
[merge\_similar\_semantics](/api-docs/prepare/transform/merge_similar_semantics)
step for more information.
## Parameters
* **Column**: the column to search and group terms in
* **Determine Language**: specify the language of your terms. You can either set it manually, or select a column that holds the value for each row's language.
* **Strength Threshold**: a factor in the $[0,1]$ range to make the algorithm more or less sensitive. A value of 1 will merge all ocurrences, while a value closer to 0 will search for stronger correlation between the terms, thus being much more strict with the merging.
# Group Similar Spellings
Source: https://docs.graphext.com/documentation/data-preparation/data-enrichments/group-similar-spellings
Group together similarly written words that may be spelled inconsistently
This transformation is useful when, for example, you have freeform text in a survey,
where people will refer to the same concept in many different ways.
This helps clean and tidy your data by providing a consistent representation of each concept.
Check out the
[merge\_similar\_spellings](/api-docs/prepare/transform/merge_similar_spellings)
step for more information.
## Parameters
* **Column**: the column to search and group terms in
* **Strength threshold**: a factor in the $[0, 1]$ range to make the algorithm more or less sensitive. A value of 1 will merge all ocurrences, while a value closer to 0 will search for stronger correlation between the terms, thus being much more strict with the merging.
# Infer Gender
Source: https://docs.graphext.com/documentation/data-preparation/data-enrichments/infer-gender
Use the first names of people to make predictions about their gender
This function allows us to guess the gender of a person based on their name.
Check out the [infer\_gender](/api-docs/prepare/enrich/infer_gender) step for
more information.
## Parameters
* **Column**: column that holds the **first names** of the people you want to guess the gender of
# Overview
Source: https://docs.graphext.com/documentation/data-preparation/data-enrichments/overview
Enhance your data easily and conveniently
Data Enrichments are a set of functions that we can apply to our data
to upsample it, provide context, remove null values and more. These functions
provide an easy way of dealing with otherwise messier data, that can then be fed
to models or used in charts for better results.
To access enrichments, go to the Wizard -> Other -> Enrichments
![Enrichments path]()
Here, we describe each of the different types of enrichment you can use.
Adjust the influence of individuals in your survey using
a variable with **predefined weights** to scale your survey data.
Use a function to fill missing values
Train a model to infer missing values
Group together similarly written words that may be spelled inconsistently
Group together words with similar meanings
Use the first names of people to make predictions about their gender
Group texts that refer to the same places
Use latitude and longitude variables to enrich your data with the Spanish
census information
# Predict Missing Values
Source: https://docs.graphext.com/documentation/data-preparation/data-enrichments/predict-missing-values
Train a model to infer missing values
This function allows us to train a model to fill in the missing values present in your data.
Check out the [infer\_missing](/api-docs/prepare/enrich/infer_missing) step for
more information.
## Parameters
* **Column** to fill values in
The model will try to fill the null values using the information already present in the column.
# Add Demographic Data for Spain using coordinates
Source: https://docs.graphext.com/documentation/data-preparation/data-enrichments/spanish-demographic-coordinate-data
Use coordinates to enrich your data with the Spanish census information
This function fetches information from the Spanish census which may be relevant
to the analysis you are working on. By providing the coordinates of your data points
we can bring useful contextual information such as age brackets, gender distribution
and more.
Check out the
[fetch\_demographics\_es](/api-docs/prepare/enrich/fetch_demographics_es) step
for more information.
## Parameters
* **Latitude**: column holding the latitude component of your data points
* **Longitude**: column holding the longitude component of your data points
# Standardize Locations
Source: https://docs.graphext.com/documentation/data-preparation/data-enrichments/standardize-locations
Group texts that refer to the same places
This function allows us to merge excerpts that refer to the same
place. This is useful when dealing with neighbourhoods or cities,
that can be mentioned in many different ways.
This step really just calls the
[merge\_similar\_spellings](/api-docs/prepare/transform/merge_similar_spellings)
step with some predefined parameters for convenience. Check it out if you need
more information.
## Parameters
* **Column**: column to standardize
* **Strength Threshold**: a factor in the $[0,1]$ range to make the algorithm more or less sensitive. A value of 1 will merge all ocurrences, while a value closer to 0 will search for stronger correlation between the terms, thus being much more strict with the merging.
# Upsample Survey Data
Source: https://docs.graphext.com/documentation/data-preparation/data-enrichments/upsample-survey-data
Adjust the influence of individuals in your survey using a variable with predefined weights to scale your survey data.
Upsampling surveys is usually a good practice when there might be under (or over) represented
groups in your data. By using a scaling factor, we can adjust the influence each individual
has in the hopes of better representing the actual distribution, had we sampled indefinitely.
Check out the [upsample](/api-docs/prepare/filter/upsample) step for more
information.
## Parameters
* **Weights**: numeric column with weights representing by how much this particular row's influence should be scaled
* **Minimum number of rows**: number of rows you need to have in your dataset to ensure that the weighted representation accurately reflects the population, even considering the least influential respondent
# Introduction to data preparation
Source: https://docs.graphext.com/documentation/data-preparation/dp-introduction
Start exploring your data and discovering insights in under 5 minutes
# Sampling data
Source: https://docs.graphext.com/documentation/data-preparation/sampling-data
Start exploring your data and discovering insights in under 5 minutes
# Finding data
Source: https://docs.graphext.com/documentation/data-preparation/sampling-data/finding-data
Search for anything within your dataset
Graphext has a lot of tools that enable you to move quickly through the interface and through your data.
There are a lot of ways to find things within your data. Let's break them down:
## Finding columns in the table
You can find where a column lives in the table by simply searching for it in this little magnifying glass, just above the table.
## Finding variables in your data
You can also find variables in the left and side bars by simply searching for them in this search field.
Pinned columns behave the same as normal, unpinned ones.
You can also reach to the variable manager, the gear icon on the top right corner, to get a bird's eye view on all your variables' metadata.
## Finding values in columns
Lastly, you can also find specific values present in your variables.
You can do this in high cardinality categorical variables and in text variables. You'll see (you guessed it) a little magnifying glass
right next to the variable's name. Click on it, search for whatever you are looking for and you'll now have it present in the crossfilter's view.
# Sample data
Source: https://docs.graphext.com/documentation/data-preparation/sampling-data/sample-data
# Dealing with nulls
Source: https://docs.graphext.com/documentation/data-preparation/sampling-data/view-nulls
Get a quick overview of how much data you are missing
It's not uncommon to have gaps in your data. Whether the measurements were not correctly made,
or there's simply missing data, it's something we should consider and spend some time on before
doing anything else with our data.
Graphext makes it very easy to spot where your data is missing and how much.
Graphext can deal with null data no problem. The table will tell you if a cell is null, like this:
![table nulls]()
Most importantly, we can get a sense of how many null rows we have in each column by simply
checking out its corresponding little cross filter on the side:
![cross filter nulls]()
The blue and white bar represents the whole dataset. The blue part indicates the currently selected
valid rows. The white, empty part refers to those rows whose value is `null` for this column.
Moreover, we can select any of these sections, which will filter the data accordingly. This is useful to
check if there's any reason behind the null values, or to get rid of all the null values altogether in
one single click.
![null selected]()
# Add dataset information
Source: https://docs.graphext.com/documentation/data-preparation/variable-management-ui-config/add-dataset-description
Adding information to your dataset is a very good practice. Adding a description of what the data
describes and a source of where you found it is always convenient for reproducibility, and to
keep everyone on the same page, even your future self.
## Adding a description and sources to your data
To add a description and a source, click the Dataset Info button,
the little i on a circle above the column headers.
![dataset metadata]()
This will expand two text fields in which you can add a description and a link to the
dataset source.
# Adding variable descriptions
Source: https://docs.graphext.com/documentation/data-preparation/variable-management-ui-config/add-variable-description
Descriptions make it much easier to understand what each variable is all about.
There are two ways to adding descriptions to a column in Graphext.
## Adding a description to a column from the data table
### Using the menu
The easiest way to enable the description view is to go to the dataset options, just above the
column names in the data table, and click "Show Variable Descriptions". This will show text
boxes under the column names. These are editable, change the description to your liking.
![Edit description
menu]()
### A quicker shortcut
Another way is to click the blue line under the column name.
When hovering with your mouse, a blue line should appear. This will expand a little
text field, in which you can write freely.
Click on the blue line again, now under the text box, to bring it back to its original position.
## Adding a description to a column from the Variable Manager
You can also add descriptions to any column by going into the Variable Manager, the gear in the
top right corner of the interface, and enabling the column layout.
You should be able to see a text area in each card displaying "Add a description". Write all the descriptions
you need and click the "Save" button in the lower right corner to save all your changes.
![Renaming a column from the variable
manager]()
# Arrange columns in the Data Table
Source: https://docs.graphext.com/documentation/data-preparation/variable-management-ui-config/arrange-columns-data-table
The Data Table in Graphext is really flexible, and allows for great manipulation. Columns
can be rearranged, which is convenient when trying to compare and contrast data.
There are a couple of ways to arrange columns in the Data Table:
## Dragging columns
Quickest and most intuitive way to put a column where
it needs to be is to simply **grab and drag it**.
## Sending a column first or last
You can also move columns directly to the first or last position, by using the column
settings menu.
![Rearrange column menu]()
This is particularly convenient when you create predictive models or use the Recipe,
processes that create new columns that are usually very interesting to analyze. These columns end up
last. To avoid having to drag the column all the way through, simply send it first and then
reorder there.
Other times, we usually send columns last as a way to *hide* them from the view. It's up to you!
## Arranging columns in the Variable Manager
Both of these processes can be performed in the Variable Manager. Drag a variable card to put it in a new
position, or send it first/last using the arrow icons in the top right corner of each card.
# Arrange variable order
Source: https://docs.graphext.com/documentation/data-preparation/variable-management-ui-config/arrange-variables-menu
The left and right sidebars display the cross filter for each of the variables
present in your data. They're sorted in the same way the original columns are sorted
when imported. We can change this order to our liking.
In order to change the order in which the variables in the left
and right sidebars appear, we must go to the Variable Manager.
From there, we can simply drag and drop them, arranging them in whatever order we
find most useful.
You can click on the 3 display buttons to the right of the search bar. This
will display the cards with different levels of detail. The right-most button
will display very small cards that only include the name, which displays much
more variables on view and makes it a lot easier to handle.
After that, click save in the lower right corner of the screen to save your changes.
# Change data table layout
Source: https://docs.graphext.com/documentation/data-preparation/variable-management-ui-config/change-data-table-layout
# Rename a variable
Source: https://docs.graphext.com/documentation/data-preparation/variable-management-ui-config/change-variable-name
There are two ways of changing a column name in Graphext.
## Changing a column name from the data table
The quickest and easiest way is to simply click on the column name itself. This field
is editable, so you can input whatever name you like most.
![Renaming a column from the data
table]()
After editing the name, a tooltip will appear when hovering your mouse over
the it, displaying the current name and the original column name when the data
table was initially imported.
## Changing a column name from the Variable Manager
You can also change the name of any column from the variable manager, in much
the same way as with the table. Simply click on the name and edit it to your liking.
![Renaming a column from the data
table]()
If the variable manager feels "locked" and you cannot edit the variables, it may be because of two things.
* You don't have edit access to the project
* You are in Variable Viewer mode, instead of the Variable Manager mode. In order to continue editing your
variables, return back to "Variable Manager" in this drop down menu on the top left corner.
![Variable Viewer Mode]()
# Add colors to the cross filters
Source: https://docs.graphext.com/documentation/data-preparation/variable-management-ui-config/color-filters
The [Cross Filters](/concepts/graphext-concepts/cross-filters), the interactive charts you can find in either sidebar
can be customized to more easily represent your data. If they correspond to a categorical
variable, you can change the colors of the different segments, or each category the variable holds.
# Customizing statistics
Source: https://docs.graphext.com/documentation/data-preparation/variable-management-ui-config/customize-stats
The [cross filters](/concepts/graphext-concepts/cross-filters) associated with a numerical column have an extra button
called "Statistics", where you can toggle a menu that shows just that: the statistics for that
column.
This can be toggled by clicking in the little x icon in the top right corner, representing the mathematical
notation for the average.
![Stats]()
In case you made a selection, statistics will be shown for both the whole
dataset and the selection made.
## Customizing the statistics shown
By default, this view shows the min, P25, avg, median (P50), P75, and max. In case you are interested in different
values, you can hover your mouse over the statistics, and a button will appear, saying "Edit stats".
![Edit Stats]()
A pop up will appear, with a list of all possible statistics shown and checkboxes for you to toggle. You can select
up to 6 of these at a time.
The complete list of metrics includes:
* min (minimum)
* P25 (first quartile)
* avg (average)
* median (second quartile)
* P75 (third quartile)
* max (maximum)
* sum
* std dev (standard deviation)
Each metric has the data for the whole dataset and the selection (if present) next to it, for easier decision making.
![Edit Stats menu]()
# Delete a variable
Source: https://docs.graphext.com/documentation/data-preparation/variable-management-ui-config/delete-variable
Deleting a variable can sometimes be useful, although you might consider [hiding it](/documentation/data-preparation/variable-management-ui-config/hide-variable) instead
in case some important or sensitive information is at play.
## Ways to delete a variable
### Deleting a variable from the Data Table
To delete a column from the Data Table, simply click on the options menu
next to the desired column's name and click "Remove Variable".
![Delete variable data
table]()
### Deleting a variable from either sidebar
You can also delete a variable by accessing the menu in the top right
corner of the variable's cards in either sidebar.
![Delete variable data
table]()
### Deleting a variable from the Variable Manager
You can also delete a variable from the Variable Manager, by accessing
the menu in the top right corner of the variable's card.
![Delete variable variable
manager]()
## Implications of deleting a variable
Deleting a variable means to completely remove it from the dataset, effectively "creating" a
new version of the dataset without it.
Upon clicking on "Remove Variable", a warning message will pop up, prompting you to confirm
whether you are sure to perform the deletion of the variable.
This message will also display all the columns that may depend on the variable you are deleting.
These variables would be deleted too.
![Deletion Warning]()
### Undoing deletions
If you deleted a variable by mistake, you can reach to the undo button in the top right corner of the interface
to undo that operation.
![Undoing deletion]()
Undoing the operation will only be possible right after deleting. If you reload the page or come to a new
session, it will no longer be possible to recover the data.
If all these things happened and you need to recover information, [reach out to the team](mailto:support@graphext.com) for help.
# Editing segments
Source: https://docs.graphext.com/documentation/data-preparation/variable-management-ui-config/editing-segments
After creating a [segment](/concepts/graphext-concepts/segments), it will be named with a sum up of the set
of filters you composed it from. This name can be edited to your liking
for easier understanding of it.
## Enabling edit mode
To edit a segment, click the Edit Properties button in the top right corner of the segment.
![Edit properties of
segment]()
## Edit the names and/or colors of each segment
This will enable Edit Mode on the segment. You can now click on any of the segments to edit them
(1), change the name of the segment (2), optionally change their color, and finally apply the changes
by clicking the apply button or hitting the Enter key (3).
![Edit properties of
segment]()
Repeat this process for any segments you want to modify. After that, click the Save button to save all
your changes.
![Edit properties of
segment]()
## Editing a categorical variable
Categorical variables behave in much the same way, or, put it the other way around: segments are a kind of
categorical (multivalued) variable we create out of a set of filters.
This means that the cross filter corresponding to any categorical variable can also be edited.
![Edit Categorical
Menu]()
![Edit Categorical
Result]()
# Enable histograms in the table header
Source: https://docs.graphext.com/documentation/data-preparation/variable-management-ui-config/enable-histograms-header
# Freeze columns in the data table
Source: https://docs.graphext.com/documentation/data-preparation/variable-management-ui-config/freeze-columns-table
Freezing (or pinning) columns to the left of the table is a useful way to keep
a set of variables always in sight.
To freeze (or pin) a column, simply head to the column menu and select Pin.
![pin column]()
This will keep the pinned column always visible.
![pin column]()
# Group variables using tags
Source: https://docs.graphext.com/documentation/data-preparation/variable-management-ui-config/group-variables
[Tags](/concepts/graphext-concepts/tags) are a powerful way of organizing your variables, ane especially useful on
datasets with lots of related variables.
## Adding tags
There are two main ways we can add tags to a variable: using the Variable Manager
or using a Step in the Recipe.
### Using the Variable Manager
Here's an example where we create the "host" tag in one variable, and then select multiple variables to apply the
tag to all of them in one step.
### As a Step in the Recipe
The step [configure\_tagged\_columns](/api-docs/report/configure_ui/configure_tagged_columns) will
allow you to tag every column passed with a tag of your choice.
```erlang theme={null}
configure_tagged_columns(ds.columna, ds.columnb, { "tag": "very important" })
```
### Using the tagged columns in the recipe
Sometimes it's useful to process all the columns belonging to a tag using a step.
You can copy all variable names belonging to a particular tag by selecting the tag itself and then clicking
the button in the upper right corner of the pop-up.
![Copy Variable Names Tag]()
In this particular case, this is what gets sent to the clipboard:
```erlang theme={null}
"host_id", "host_url", "host_name", "host_since", "host_location", "host_about"
```
Since steps that work with groups of variables usually need them listed like this, you can simply
paste this into your step's options and quickly get going.
# Hide a variable
Source: https://docs.graphext.com/documentation/data-preparation/variable-management-ui-config/hide-variable
Hiding a variable removes it from the data table or the layout, without the need to delete it completely.
There are a couple ways to hide a variable.
## Hiding a column from the data table
In order to hide a column from the data table, click on the options icon next to its name,
and select "Hide". The variable will disappear from the data table, but not from the sidebars.
![Hiding a variable from the data
table]()
## Hiding a variable from the sidebar
In order to hide a variable from either the left or the right sidebar, we must
use the Variable Manager, which is the gear icon located at the top right corner of the interface.
### Step 1: Click on Manage Variable
A quick way to hide a specific variable, is to click on the options icon in the variable
card, and select "Manage Variable".
![Hide variable side
bar]()
### Step 2: Hide the variable from the sidebar
This will automatically open the Variable Manager and position the view in the variable we
are interested in. The rest of the variables will appear dimmed for a moment. From here, you can
hide the variable from the sidebar by clicking on the little eye icon on the top right of the variable's
card.
When you are done, click the "Save" button in the lower right corner. This will save the layout you've
created.
![Hide variable side
bar]()
# Overview
Source: https://docs.graphext.com/documentation/data-preparation/variable-management-ui-config/overview
Essential tooling to move through Graphext
Organizing and managing your variables is a key aspect when dealing with lots of columns, or
variables. Here are some commonly used actions that will help you deal with your data in an
easier and more efficient way.
## Variable management
Move, edit, delete and deal with the variables in your project.
## Data Table Management
Change the order in which the columns appear
Sort the table with respect to a column
Pin a column to the left so it's always visible
## UI Customization
Use tags to make sense of your variables and operate more quickly over groups
Pin a variable to the left sidebar so it's always within reach
Sort the variables in the left and right sidebars to your liking
## Dataset Metadata
# Pin variables
Source: https://docs.graphext.com/documentation/data-preparation/variable-management-ui-config/pin-variables-menu
Pinning variables to the left sidebar allows you to keep them always visible to the left,
avoiding having to constantly search for them.
To pin a variable, head to the options button in the upper right on the variable's card and
click Pin Column.
![Pin Menu]()
After that, the column will now appear in the left sidebar, always visible and within reach.
![Pinned Column]()
# Sort the Data Table
Source: https://docs.graphext.com/documentation/data-preparation/variable-management-ui-config/sort-data-table
As any table you might have dealt with, the Data Table can be sorted by any given column,
given that the column has a way to be ordered.
To do this, simply reach out to the column menu and go to Sort. Then choose whether you want
ascending or descending order.
![Sorting the table]()
This will sort all the rows based on the columns ordering.
For columns that do not present a clear ordering, sorting will be done
alphabetically. This applies to `category`, and `text`.
If you have a categorical variable that has a clear ordering but it doesn't
sort properly, check how to convert it to an **ordinal variable** by
[specifying the ordering in categorical
variables](/documentation/data-preparation/variable-management-ui-config/specify-order-in-column).
Examples of this kind of variables are **shirt sizes**, where "S" is smaller than "M", which is smaller than "L", or
**survey answers** like "Disagree" which is smaller than "Don't Mind/Don't Know", which is smaller than
"Agree".
# Specifying order in categorical variables
Source: https://docs.graphext.com/documentation/data-preparation/variable-management-ui-config/specify-order-in-column
It is common to have a categorical column whose values have a clear order. This can
be shirt sizes, like "S", "M", "L" or "XL". The category "L" is *greater* than "M".
## Applying ordering to the values
To do this, we reach out to the column menu and select the **Order** option.
![Menu]()
All the different values for this column will appear, with a handle on the left.
Drag and sort the categories in the order you want, in **descending order**.
Bigger elements first.
![Menu]()
Hit save and you are good to go!
## Using the recipe
We can also order categories through the
[Recipe](/concepts/graphext-concepts/recipe), writing a
[step](/concepts/graphext-concepts/steps).
### Intro and Resources
To do this, we are going to use the [order\_categories](/api-docs/prepare/transform/order_categories) step.
You can follow [this example](https://app.graphext.com/projects/UHJvamVjdC05ODA3MA==/v/data).
A video tutorial is available to do this exact process in the Titanic Data set.
You can watch it [here](https://youtu.be/kM9KCq5K2nw?t=239\&si=-CG2PA6UVZS8kAAE) in case it makes it easier to follow.
### Open the Recipe
Reach out to the little scroll icon in the top right corner.
![Recipe icon]()
### Enable Code Mode
This will allow us to see and edit the code.
![Code Mode in recipe]()
### Search for the order\_categories step
Look for a line of code that looks like this:
```erlang theme={null}
create_project(ds)
```
and write a **new line ABOVE it**. It is very important that you keep this in
mind. Our step won't have any effect if it is written after the `create_project` step.
Start writing `ord`, and the list will show the `order_categories` step.
Press enter, tab or click on the suggestion to accept it.
![Order Categories
suggestion]()
This operation will write this piece of code on the same line:
```erlang theme={null}
order_categories(ds.input) -> (ds.output)
```
### Edit the step with your column name
Because the column we want to take the information from is called `size` (careful, it's case sensitive!),
we change the code so it looks like this:
```erlang theme={null}
order_categories(ds.size) -> (ds.size)
```
This gives an error because we are trying to create a column that already exists, `size`. We can either
create a new column with the ordering applied, or we can **overwrite** the existing column to include
the ordering information.
To create a new column, simply change the output from size to any other name, for example:
```erlang theme={null}
order_categories(ds.size) -> (ds.size_ordered)
```
To overwrite the colum, we use the overwrite operator, like this:
```erlang theme={null}
order_categories(ds.size) => (ds.size)
^ this changed from - to =
```
### Apply the ordering
Let's apply the ordering. To do this, edit the step like this:
```erlang theme={null}
order_categories(ds.size,
{"categories": ["XXS", "S", "M", "L", "XXL", "XXXL"]}
) => (ds.size)
```
This involves creating an options object with `{}` that has a field `categories`. This field expects a list
of all the categories present in this column **in ascending order**. "Smaller" elements first.
You can basically copy this piece of code and substitute "size" for the name
of your column, and the categories list by all the different categories just
like here: each in between quote marks and separated by commas.
### Done!
Now, we click "Run" in the lower right corner to save.
We can now see that our variable Size has a sorting icon next to it, allowing us to [sort the whole table](/documentation/data-preparation/variable-management-ui-config/sort-data-table).
![Size sortable]()
# Choosing a chart
Source: https://docs.graphext.com/documentation/data-visualization/choosing-a-chart
## Interface Overview
The **Chart Builder** is designed to facilitate the creation of various chart types for data visualization. Below is a breakdown of each element present in the interface:
![Interface Overview]()
### 1. Select Variables to Analyze
* **Search Bar**: Allows you to quickly find and select the variables you wish to analyze. You may type the name of the variable into the search field.
* **Add Button (+)**: This button, located on the right side above the chart options, allows you to add additional variables to the analysis.
### 2. Choose a Chart Section
This section provides a gallery of chart types to choose from. Each chart option comes with a representative icon and is selectable to create that specific type of visualization.
Each chart comes with a set of more complete variants. To explore the different variants available for each chart type, simply hover your cursor over the specific chart's card. Each variant is symbolized by a unique icon. For instance, when you hover over the 'Bar Chart', you'll see variants like 'Stacked Bar Chart' or 'Grouped Bar Chart'.
## Two ways of creating a chart
There are two different ways to creating a chart:
* **Guided**: when you have identified the variables you wish to visualize, but haven't decided on the specific chart type.
* **Off-road**: when you have a specific chart type in mind, and you want assistance in the selection of variables.
### Guided way to create a chart
To begin, locate the input at the top and identify the variable(s) you wish to analyze.
You can choose to focus on one or multiple variables, with the option to add more using the 'Add Button'.
Upon introducing the variables, Graphext will automatically generate a chart recommendation.
This is based on several factors such as the type of variables, cardinality, and distinct characteristics (for instance, whether the numbers are discrete or continuous, or if the categories have a predefined order).
These recommendations will be displayed immediately on the cards.
The most suitable chart variant will be prominently displayed, while the variants
incompatible with the selected variables will be disabled.
You can determine why a specific chart is disabled by simply hovering over it.
Once users have selected their preferred chart, Graphext will proceed to compute the optimal mapping of variables to slots. This relies on a set of heuristics that evaluate all possible configurations. For instance, a simple heuristic might be: a variable of the 'Date' type will perform better on the X-axis than a categorical variable.
### Off road way to create a chart
Using the "Off-road" method is particularly useful when you are well-acquainted with
the type of chart that best represents your data and the story you want to tell.
This method can often be faster as it bypasses the variable selection step and goes straight
to the chart creation, allowing you to shape your data visualization with more direct control
from the outset.
You can select the chart type that best fits the visualization you have in mind for your data.
Graphext will analyze the variables in the the dataset and select those that yield the most
legible configuration of the chart. It tries to make a chart that's easy to read and
understand. You can then directly select the variables that are interesting for them
in the proper mapping slot.
# Compare segments
Source: https://docs.graphext.com/documentation/data-visualization/compare-segments
Start exploring your data and discovering insights in under 5 minutes
# Correlations and patterns
Source: https://docs.graphext.com/documentation/data-visualization/correlations-and-patterns
Start exploring your data and discovering insights in under 5 minutes
# Create plots and charts
Source: https://docs.graphext.com/documentation/data-visualization/create-a-plot
Create publish-ready charts in minutes
## Interface Overview
The chart builder interface is designed to facilitate the creation of various chart types for data visualization. Below is a breakdown of each element present in the interface:
![Interface Overview]()
### 1. Select Variables to Analyze
* **Search Bar**: Allows you to quickly find and select the variables you wish to analyze. You may type the name of the variable into the search field.
* **Add Button (+)**: This button, located on the right side above the chart options, allows you to add additional variables to the analysis.
### 2. Choose a Chart Section
This section provides a gallery of chart types to choose from. Each chart option comes with a representative icon and is selectable to create that specific type of visualization.
Each chart comes with a set of more complete variants. To explore the different variants available for each chart type, simply hover your cursor over the specific chart's card. Each variant is symbolized by a unique icon. For instance, when you hover over the 'Bar Chart', you'll see variants like 'Stacked Bar Chart' or 'Grouped Bar Chart'.
## Two ways of creating a chart
There are two different ways to creating a chart:
* **Guided**: when you have identified the variables you wish to visualize, but haven't decided on the specific chart type.
* **Off-road**: when you have a specific chart type in mind, and the selection of variables to be used will be made subsequently.
### Guided way to create a chart
To begin, locate the input at the top and identify the variable(s) you wish to analyze.
You can choose to focus on one or multiple variables, with the option to add more using the 'Add Button'.
Upon introducing the variables, Graphext will automatically generate a chart recommendation.
This is based on several factors such as the type of variables, cardinality, and distinct characteristics (for instance, whether the numbers are discrete or continuous, or if the categories have a predefined order).
These recommendations will be displayed immediately on the cards.
The most suitable chart variant will be prominently displayed, while the variants
incompatible with the selected variables will be disabled. Please note that you can determine
why a specific chart is disabled by simply hovering over it.
Once users have selected their preferred chart, Graphext will proceed to compute the optimal mapping of variables to slots. This relies on a set of heuristics that evaluate all possible configurations. For instance, a simple heuristic might be: a variable of the 'Date' type will perform better on the X-axis than a categorical variable.
### Off road way to create a chart
Using the "Off-road" method is particularly useful when you are well-acquainted with
the type of chart that best represents your data and the story you want to tell.
This method can often be faster as it bypasses the variable selection step and goes straight
to the chart creation, allowing you to shape your data visualization with more direct control
from the outset.
It is very straightforward. You can select the chart type that best fits
the visualization you have in mind for your data. Graphext will analyze the variables
in the the dataset and select those that yield the most legible configuration of the chart.
It tries to make a chart that's easy to read and understand. You can then directly select
the variables that are interesting for them in the proper mapping slot.
# Customizing Axes
Source: https://docs.graphext.com/documentation/data-visualization/customizing-axes
Get new perspectives on your data
In the same way you can [customize the looks on your chart](documentation/data-visualization/customizing-charts), you can also customize its different axes.
These are not aesthetic changes, though, as one can tell completely different stories by changing these settings.
## Axes configuration
Depending on the nature of the variable used for a given axis and the type of chart you are working
with, some options will come up to help you shape the data and resulting visualization.
![Axes config general]()
Let's review some of the common ones:
### Temporal Axis
![Temporal Axis]()
If the variable has a time component, the axis configuration will offer some options to aggregate data with
different "resolutions". The smaller the time unit, the finer the visualization.
Particularly for dates, some semantically sensible options are given, such as grouping for each year,
week or quarters, among others.
### Numeric X axis
![Numeric Axis]()
If the X axis is numeric, binning options are offered based on the extent (minimum and maxium) the column
has. "Binning" means to break down the whole range of values in intervals, like *chunks*, and then seeing
how many rows lie in that interval. We can then make aggregations on each of these groups, like counting, averages, medians,
and such.
### Numeric Y axis
![Numeric Y Axis]()
If the Y axis is numeric, aggregation options like average, median or standard deviation are offered for you to explore.
### Categorical Axis
If the axis is categorical, we have two options to customize:
We can limit the number of categories and choose only the $n$ "top" or "bottom" values for that category.
This helps in reducing potential noise for categories that don't have much relevance in your visualization.
![Categorical
Axis]()
We can also sort these categories by some criteria, like the X axis metric, selection, ordinal or alphabetical order.
![Categorical Axis]()
In general, this behavior is shared across the board, with some notable exceptions, like [Box plots](https://localhost:3000/documentation/data-visualization/types-of-chart/box-plot#axes-configuration).
All particular cases are discussed in their corresponding page.
### Number of rows
The option "number of rows" appears when you can have a sum up of a numeric value, like an average, sum or median.
With the number of rows, some options arise that are particularly interesting when searching for patterns. This can
all be expressed as relative comparisons between categorical values.
The most common, and default one, is the count, where we literally express the number of rows. But other operations can
be made.
![New aggregation methods for
categories]()
#### Relative Count (All)
This method changes the scale to percentage, where now, instead of showing how many rows fall into these categories,
we show the percentage of rows.
In this example, we can see the proportion of gender in each age bracket. The left-most blue bar indicates that there
are \~89.6 transactions made by 18 – 24 years old women, which corresponds to 5.56% of the whole dataset.
These charts are interactive, use your mouse to explore!
#### Relative Count (Color)
The relative count by color changes the distribution so that each color must add up to 100%. This lets us know where a
particular segment is most over (or under) represented.
This answers the question "what's the age distribution on women?"
For example: the tallest red bar indicates that most of the people that didn't want to respond are between 25 – 34 years old.
#### Relative Count (X)
The relative count by X axis changes the distribution so that each individual segment adds up to 100%, showing a kind of
local distribution per each segment.
This answers the question of "what's the gender distribution on people in the 18 – 24 age bracket?"
For example: among all the people in the 35 – 44 age bracket, 55% are women and 43.9% are men.
#### Cumulative Sum
The cumulative sum option allows you to see the rate of change of the number of rows between a given segment and the
next.
This answers the question "how many men under 54 years do we have in our data?" which corresponds to the fourth
orange bar in the example: 577K.
# Customizing Charts
Source: https://docs.graphext.com/documentation/data-visualization/customizing-charts
***
Plots can generally be highly customized. In any plot, click on the Customize button on the top right corner.
![Customize Button]()
This will open a menu with a myriad of options to customize your plot.
![Customize Options]()
## Filtering
While not immediately obvious, all the [cross filters](/concepts/graphext-concepts/cross-filters) you have currently
applied will actually show up in your plot, in real time. You can switch
to the Data tab, hone in to a thin slice of your dataset, and come back to the
Plot tab and continue editing your chart. All progress will be saved.
By toggling the Customize button, you can swap between the customize panel and
the cross filters panel. This will allow you to filter the data while seeing
how the changes affect your chart in real time.
Going to another tab like Data or Insights will not reset the state in Plot.
You can navigate away safely for quick adjustments.
## Themes
Themes are color and typography settings that change the look and feel of your chart. We currently
include 7 themes inspired by the most used tools and their aesthetic decisions:
* Graphext Dark
* Graphext Light
* FiveThirtyEight
* ggplot2
* Google Charts
* PowerBI
* Urban Institute
![Theme Selection]()
Example of a simple chart with all themes applied.
![light]()
![light]()
![light]()
![light]()
![light]()
![light]()
![light]()
## General Tab
The General tab holds options for changing axis names, scales, customize color segments, spacings and such.
On top of that, each plot will also display some exclusive options that may only make sense in that type of
plot.
### Visualization & Orientation
These options are generally available in multi variable charts, like line, area, and bar charts.
The **Visualization** option allows to include more variables to compare, taking you from a simple bar chart to
a more rich grouped bar chart or stacked bar chart, for example.
The **Orientation** option allows to swap the X and Y axes to make the chart horizontal or vertical. Horizontal
charts are useful when the X axis has very long names in each tick.
![Visualization/Orientation]()
### Axes Scale and Title
Axes customization of titles and scales are also generally available, where it applies. You can customize
the Y scale, and titles for each axis. If available, the color display name will change the title above the
legend for the colors.
![Axes Customization
Options]()
### Colors
When available, you can customize the color of each segment and even change the order in which they appear.
## Layout tab
The Layout tab remains mostly the same in every chart type. Here you can change the size and proportions
of the *canvas* the chart is over.
You can also add a title, a subtitle, a description and a footer to your chart, essential when publishing
anywhere.
![Layout Tab]()
## Annotations tab
The Annotations tab allows you to add little pieces of information, like value labels and statistic marks.
This tab and the behavior of the marks will change depending on the orientation and nature of your chart.
![Annotations tab]()
### Free Text Annotations
Free text annotations allow you to position any given piece of text wherever you want. This makes it easy to
emphasize some aspects of your chart that may be overlooked.
### Highlight shape
Additionally, you can add an arrow that may be attached to some text. This is very powerful to clarify something
in your chart but just putting the text is not clear enough.
When you add a new highlight shape, an arrow will appear in the middle of the chart. You can simply move the endpoints
wherever you like. Write its corresponding text and move it where it best looks.
You can configure the color, stroke style, endpoint (head) and width of the arrow.
You may also change the color, size and style (italic, bold) and alignment of the associated piece of text.
![]()
### Reference Marks
Reference Marks are labels that have an associated value with respect to a given axis. To put one, click the +
icon and select the axis in which you want to put your mark. Depending on its nature, you'll be prompted to select a
specific value to position it. You can also write a label that offers more information around that value.
![Reference Marks]()
On top of that, you can also select the "type" of value, which can also depend on the nature of your axis. For numeric or
date values, an arbitrary "Constant" is offered alongside some metrics like minimum, maximum, average and more.
For categorical variables, a dropdown is offered listing all the possible categories.
### Show Values
Shows the value for each element in your chart, making comparisons that much easier and convenient.
You can change the formatting and positioning.
![Show values]()
## Specific options
As previously mentioned, there are some specific customization options to each chart. For example, only the
bubble chart has the option to scale up or down the size of each data point.
You may check the specific options for each chart under their corresponding page.
# Exporting charts
Source: https://docs.graphext.com/documentation/data-visualization/exporting-charts
After creating and customizing your chart to your liking, the next step is to share it with the world.
Graphext offers several options to do so in handy ways.
You can export charts as PNG, SVG, Embeds and even CSV.
## Exporting charts
Once you are done with your chart, click on the Export Chart button, next to the Customize one.
![export button]()
This will open up a helper menu to assist in the process:
![export button]()
Let's review each field:
### Name
Lets you change the final filename that will be downloaded.
### Format
Lets you choose from the available formats.
PNG and SVG download the corresponding raster or vector file.
Embeds can be copied and pasted in webpages. You can see
examples of them in action in the [Types of Charts section](/documentation/data-visualization/types-of-chart/overview).
CSV generates a file with the current aggregation underlying the chart you just created.
### Size & Options
Width will scale up the image to be bigger, but proportional to what you see in the editor.
If the image is a PNG, it makes it proportionally bigger, for better resolution. In the case of SVG,
it simply sets the width and viewBox properties acccordingly.
Padding is used to create an empty frame around all elements of the chart to allow it to breathe and
not appear crowded or busy.
You can optionally enable Transparent background for easier post-processing and including the Graphext watermark.
# Area Chart
Source: https://docs.graphext.com/documentation/data-visualization/types-of-chart/area-chart
***
Area Charts behave a lot like line charts and bar charts combined. While at first glance
they can feel quite similar to a curved line chart, they are better suited to show
differences between distributions of data rather than trend evolution.
For example, in this dataset, we have a list of transactions that can either be
Income or Expenses.
Making an area chart gives a visual perspective of what the net benefit (income - expenses) is,
by visually subtracting the blue area to the orange one. Check it out live [here](https://dev-embeds.graphext.com/ef129709dac87e76/index.html?section=data).
In Graphext, you have 4 subtypes of line charts to choose from:
* Simple Area Charts
* Stacked Area Charts
* Stacked Relative Area Charts
* Segmented Area Charts
## Simple Area Chart
Old classic area chart, two variables to rule them all.
## Stacked Area Charts
As exposed in the example above, multiple area charts allow for very easy comparison
of distributions.
## Stacked Relative Area Charts
Staked Relative area charts normalize all their values to add up to 1. This fills the
whole Y scale, and tells you the percentage each X value occupies out of the total.
This example is the same one as above but now stretched to fill
the whole 100% range. Now we can appreciate how much each season contributes to the
overall yearly temperature in relative terms.
## Segmented Area Charts
Segmented area charts behave in much the same way as [Segmented Line charts](/documentation/data-visualization/types-of-chart/line-chart#segmented-line-charts) do.
You get a different plot for each category displayed.
This example is the same as the one in the beginning, but separating each Wage
bracket into its own little plot.
## Customizing Area Charts
### Color
Color customization for area charts works in the same way as with any chart.
You can learn more here: [Customizing colors in a chart](/documentation/data-visualization/customizing-charts#colors).
### Interpolation
Interpolation customization for area charts works in the same way as line charts. You can
learn more here: [Customizing interpolation for line charts](/documentation/data-visualization/types-of-chart/line-chart#interpolation).
# Bar Chart
Source: https://docs.graphext.com/documentation/data-visualization/types-of-chart/bar-chart
***
Bar Charts are, arguably, one of the most popular types of chart. They represent relationship
between variables in a very clearly visual way, allowing to compare the height of
each bar. This makes it very easy to spot changes in your data.
There are 5 kinds of bar charts available in Graphext:
* Simple Bar Chart
* Grouped Bar Chart
* Stacked Bar Chart
* Stacked Relative Bar Chart
* Segmented Bar Chart
## Simple Bar Chart
We know how this goes.
A particularly useful, albeit common, use of bar charts is using one axis to measure the number of
ocurrences in your data. In this example, we count how many of the credit card transactions
belong to a given category and we plot that number.
## Grouped Bar Chart
Grouped bar charts can display all the combinations between the values of two
variables. This allows for a great way of providing a ton of information at a glance.
We can see in this example that we have a variable "**Overall skill**" which holds the values
"bad", "mid", "good" and "great", and another variable "**Wage bracket**", which holds the
values "low", "mid-low", "mid-high" and "high". This chart presents the number of players
that lie in each of the 16 possible combinations in a very tidy format.
## Stacked Bar Chart
Stacked bar charts share a lot of similarities with grouped bar charts, with the only
difference that, instead of laying the bars laterally, they are stacked up.
The former is better to distinguish small differences in groups, the latter is
better suited for a broader perspective on the relatioship between the groups.
## Stacked Relative Bar Chart
The next logical step is, of course, normalizing the values so that they add up to 1.
This gives an even better representation of the relationship in size between each group.
We can see how this vaguely resembles a [stacked area chart](documentation/data-visualization/types-of-chart/area-chart#stacked-relative-area-charts), albeit a
bit coarser in nature.
## Segmented Bar Chart
Segmented bar charts behave in much the same way as [Segmented Line charts](/documentation/data-visualization/types-of-chart/line-chart#segmented-line-charts) do.
You get a different plot for each category displayed.
## Customizing Bar Charts
Bar Charts share much of the same functionality with [line](/documentation/data-visualization/types-of-chart/line-chart) and [area](/documentation/data-visualization/types-of-chart/area-chart) charts.
### Bar Gaps
The spacing in between each bar can be customized through the Spacing option.
![Barchart
Customization]()
Adjust the slider to make the bars thinner or larger.
Moreover, the spacing on either side of the chart can also be adjusted, in
case the proportions play out a bit more nicely.
### Color
Color customization for bar charts works in the same way as line charts. You can
learn more here: [Customizing color for line charts](/documentation/data-visualization/types-of-chart/line-chart#color).
# Box Plot
Source: https://docs.graphext.com/documentation/data-visualization/types-of-chart/box-plot
***
Box plots, or also known as whisker plots, are great at summarizing distributions across categories in your variable.
They represent the most important splits along a given distribution in your data: the [quartiles](https://en.wikipedia.org/wiki/Quartile).
In Graphext, we also represent values that extend past the [interquartile range](https://en.wikipedia.org/wiki/Interquartile_range), showing the spread that occurs above
and below it.
![Box plot extend]()
These kind of charts are quite flexible since they can display a wide range of types in an easy and information dense
format.
Box plots are not very customizable
## Customizing Box plots
Box plots are relatively simple charts on their own, and, as such, not much configuration can be made apart from
coloring the bins and changing the axes settings.
### Color
Color customization for box plots works in the same way as with any chart.
You can learn more here: [Customizing colors in a chart](/documentation/data-visualization/customizing-charts#colors).
### Axes configuration
When customizing a box plot, the dependent variable will have custom options to represent different sections of the
whole distribution, such as the whole range of values, including outliers, or statistically important sections of it.
![box plot y config]()
# Heatmap
Source: https://docs.graphext.com/documentation/data-visualization/types-of-chart/heatmap
> *Looks like I've been digging Vulfpeck forever even though it's only been
> since 2021. It appears as though I'm much more interested in Antonio Lizana
> now, which I would agree with. I guess I cannot argue with the data.*
>
> – Jesús
***
Heatmaps show proportions of occurrences in your data in a grid, where each cell indicates visually how
the proportion for that particular intersection of categories looks like.
This relationship can be customized in Graphext, configuring the Cell count.
## Customizing Heatmaps
### Cell count
The cells in the heatmap can represent several types of aggregations of your data.
![Cell Options in heatmaps]()
Let's analyze the different options with the same example as the one in the header.
#### Count
Count gives the absolute number of rows that correspond to any given combination of categories
of the two variables used. The top-left corner of this table displays `131`, which corresponds to
the number of rows that both have their year as `2020` and artist as `Vulfpeck`, after all the filters
have been applied.
We can see that's easy to spot important numbers as cells "light up" more proportionally to the number they hold.
#### Relative Count
Relative count normalizes the visible values to add up to 100. That is, the "bluest" cell, where there
was a 1.35K before, means that this particular combination corresponds to 3.13% of all streams in
the dataset.
#### Relative Count (X)
However, sometimes it is much more interesting to break down percentages per each axis.
The relative count on the X axis shows the percentage with respect to the total
number of rows that fall in each category in the X axis, and then calculates the percentage.
This means that now we are answering the question: "out of all the streams made in a
given year, how many were towards a given artist?".
For example: out of all the streams I made in 2021, \~33% were listening to Vulfpeck. Out of all the streams I
made in 2024, \~53% were listening to Antonio Lizana.
#### Relative Count (Y)
The relative count on the Y axis, does the opposite: shows the percentage with respect to the total
number of rows that fall in each category in the Y axis, and then calculates the percentage.
This answers the opposite question: "out of all the streams made towards a given artist, how
many were made in a given year?".
For example: \~63% of the streams I made to Jamiroquai happened in 2021.
# Line Chart
Source: https://docs.graphext.com/documentation/data-visualization/types-of-chart/line-chart
***
Line charts are the most popular type of chart for visualizing change over time and detecting temporal
patterns. While not exlusively used for that, it's where they really shine.
In Graphext, you have 4 subtypes of line charts to choose from:
* Simple Line Charts
* Multiple Line Charts
* Segmented Line Charts
* Seasonal Decompositions
## Simple Line Charts
The simplest form of a line chart. Shows the progression of one variable over the other.
## Multiple Line Charts
Multiple Line charts allow to see how different variabes change over time, offering a broader
perspective. Again, these are usually used to show how multiple trends change over time.
They take one more variable, which would be mapped to a new line with a new color.
## Segmented Line Charts
Segmented Line charts also show multiple variables but do so in separate charts. These are also known as
**faceted line charts**, or **faceted charts**, in general.
These are useful when you want to measure change on different variables but each variable doesn't necessarily
respond to the same Y scale. Think a computer resources dashboard, displaying CPU usage, RAM and Network.
## Seasonal Decompositions
Seasonal Decomposition charts are a type of Segmented chart that displays the different temporal components your
data presents, such as trend and seasonality.
We can see in this example from measurements of the temperature of Madrid from 1920 to 2022, how the original data
can be decomposed into its trend and seasonality components. The trend has been rising steadily, but we can see a
particular bump around 1975 and onwards.
Seasonality is also quite descriptive of the 4 seasons that occur, even when considering a 10 year window.
### Expanding charts
In the Seasonal Decomposition Charts you can expand each of the individual plots to a full-scale
version of it, for increased clarity.
## Customizing a line chart
![Customize Line Chart]()
### Color
Color customization for line charts works in the same way as with any chart.
You can learn more here: [Customizing colors in a chart](/documentation/data-visualization/customizing-charts#colors).
You can change the color of each line by either selecting a color palette for all
lines, or changing a specific category for a more semantically accurate color.
For example, in here we can have a yellow summer and a brownish
autumn, which makes it that much easier to identify at a glance.
You can change a specific color by clicking on the colored circle next to the desired
category.
When selecting a specific color for a category, this decision will take over
any color palette/theme choice. That is: the colors you set manually will
prevail over any other way of changing colors.
### Line thickness
You can change the line thickness and style of one or more lines, in the need to create a bit of emphasis
in a specific segment.
If you are dealing with a [simple line chart](#simple-line-charts), options to change the thickness and dash pattern will appear under
the interpolation section.
![Simple line chart thickness controls]()
If you are working witha a [multiple line chart](#multiple-line-charts) though, more options are available to you.
![Multiple line chart thickness controls]()
The first section of the controls remain the same: you can change the width and dash pattern and this will apply
to **every line in your chart**.
When toggling "Manage line properties", you can select a subset of segments within the category mapped to color,
that will respond to the controls underneath. This way, you can select the segments you are most interested in and
give them a different styling.
### Interpolation
Interpolation on all Line Charts can be changed between these modes:
#### Linear Interpolation
Classic straight-line interpolation between any pair of points.
![Linear Interpolation]()
#### Curve Interpolation
Defines a curve between any pair of points, making the overall result look smoother.
You can choose between Monotone, Cardinal and Natural interpolation.
While these can yield pleasing curves, they may not represent your data
faithfully. Monotone interpolation is the one that best fits the data points
while making a smooth curve.
![Monotone Interpolation]()
![Cardinal Interpolation]()
![Natural Interpolation]()
#### Step Interpolation
Step interpolation is, basically, no interpolation. It creates sharp corners
and straight vertical lines between the points. These are useful when it makes
no sense to interpolate between two points, but just want to see the difference
between them.
Step has three modes: Before, Middle and After, which define the anchor point with
respect the actual data point.
![Step Before]()
![Step Middle]()
![Step After]()
### Seasonality Decomposition: Common Scale
In Seasonality Decomposition Charts you can enable a little check box at the very bottom that
toggles between the season and the residue charts having a common scale.
![Common Scale in Seasonality
Decomposition]()
# Overview
Source: https://docs.graphext.com/documentation/data-visualization/types-of-chart/overview
Creating a chart in Graphext can be extremely easy, from just a few clicks, while also allowing for a great
deal of customization for a highly specific visualization of your data. This allows to answer easy questions
quickly, and hard questions easily.
In this section, we'll go over all the features every plot has in common, to give a broad overview of what's
possible. You can then choose to dive deeper on any particular chart type to see what's available for it.
## Types of plot
Currently, Graphext supports 6 types of charts:
More detailed instructions and nuances are addressed in each chart type's page.
# Scatter Plot
Source: https://docs.graphext.com/documentation/data-visualization/types-of-chart/scatter-plot
![Scatter plot]()
***
Scatter plots display points that are *scattered* throughout the layout, according to the two variables
we are displaying. This makes it very easy to spot **correlations** in your data; i.e when a variable
changes based on the changes of another.
There are 4 kinds of scatter plots available in Graphext:
* Scatter Plot
* Colored Scatter Plot
* Bubble Chart
* Bubble Colored Chart
## Scatter Plot
The simplest form of scatter plot. Map datapoints in two dimensions.
This example makes it easy to understand how they work. We plot a little dot for
each football player. The position of the dot depends on their weight and height. This
gives a very clear picture on how the two variables are related. If the player is taller,
it is also generally heavier, as one might expect. More mass weighs more.
![Simple Scatter Plot]()
## Colored Scatter Plot
A colored scatter plot goes a step further and takes in another variable into consideration,
using color to display it.
Here we have the same example, but now dots are colored based on the overall skill of the player.
We can see that the best players are neither too short nor too tall; and something similar goes
with the weight. Excellence lies in the middle. Aurea mediocritas.
![Colored Scatter Plot]()
## Bubble Chart
Bubble Charts, on the other hand, deal with the size of the dot to represent another variable, instead of color.
Here we can see that larger dots tend to cluster around where the score for shooting and passing is better,
giving us a hint on how these two skills affect the salary of a given player.
![Bubble Chart]()
## Colored Bubble Chart
Finally, the last logical step would be to also add color to the bubble chart, representing 4 variables
in one sitting.
We can see in this last example that even though the two skills can bring you more money, not all **great**
players earn the same. And, on top of that, being "great" is not only measured from your shooting and
passing skills, as one would expect.
![Colored Bubble Chart]()
## Customizing Scatter Plots
Scatter Plot customization is a bit different than the rest of the charts. Scatter plots allow you to change
dot size, opacity, and color, if available.
### Dot Size & Opacity
The dots' size can be customized through the Size slider, if available.
Their opacity can also be tweaked, useful when we have a lot of overlapping between many data points.
Lower opacities allow us to see where datapoints cluster more.
![Customize Size]()
### Color
Color customization for scatter plots works in the same way as with any chart.
You can learn more here: [Customizing colors in a chart](/documentation/data-visualization/customizing-charts#colors).
### Annotating Scatter Plots
![Annotating Scatter
Plots]()
On Scatter Plots, there are some unique options available:
* Show regression line, which fits a line with the least squared error
* Show the [R-Squared value](https://en.wikipedia.org/wiki/Coefficient_of_determination), that will appear in the footer of the charta
* Show the [Pearson Correlation Coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient), next to the R Squared value.
![Annotating Scatter
Plots]()
# Graphext 101
Source: https://docs.graphext.com/documentation/getting-started/quickstart
Get to know the tool with a practical example
We've prepared a deeper dive into the Titanic Dataset, showing you some cool tips and tricks on
how to interpret data and how to use Graphext to extract information quickly and efficiently.
* 00:00 Intro
* 00:26 Create a new project
* 01:03 Quick Overview
* 01:42 Step 1: Preparing our data
* 01:51 Pin Survived for easy access
* 01:56 Fixing the Pclass column
* 02:09 Rename the column
* 02:17 Cast the column
* 02:37 Transform the column using AI
* 03:42 Variable Manager
* 03:59 Using the Recipe to add ordering to the Class variable
* 06:56 Extracting the title of the person from their Name using AI
* 07:42 Step 2: Visualizing our data
* 07:52 Quick overview and analysis of the data
* 09:39 Using Plot to visualize the data
* 10:20 Customizing the chart
* 11:28 Downloading the chart
* 11:47 Saving an insight of the chart
* 12:29 Using Compare to get a bird's eye view
* 13:45 Step 3: Modelling and Clustering
* 14:00 Creating a predictive model and cluster
* 16:03 Interpreting the model
* 17:11 Interpreting the graph
* 19:45 Outro
# What is Graphext?
Source: https://docs.graphext.com/documentation/getting-started/what-is-graphext
The fastest visual analytics tool. Answer questions at the speed of thought.
Graphext is a fast interface built to answer easy and complex questions about your data.
With wide format support and extensive integrations, you can upload data from anywhere,
in any format, and immediately get answers.
## A quick walkthrough
Watch this small video to get a glimpse of what’s possible to do in Graphext in 10 minutes.
# Airtable
Source: https://docs.graphext.com/documentation/import-and-export/data-connections/airtable
Connect Graphext to Airtable and be able to bring your data and write back to your data warehouse recurrently
![]()
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Head to your Airtable and open the dataset you want to import. From the URL you need to extract the **Base ID** and the **Table ID**
* The Table ID is the first part of the URL and starts with "app..."
* The Base ID id the second part of the URL and starts with "tbl..."
![]()
To obtain the Access Token you need to navigate to [https://airtable.com/create/tokens](https://airtable.com/create/tokens). Click on "Create new token", define the name the scope and the access. At least you should include the two scopes to read (data.records:read and schema.bases:read) and give access to the table you want to import
![]()
Copy the generated token
![]()
Finally you need to include the **Table ID** the **Base ID** and the **Access Token** in the Graphext configuration UI
![]()
If you want Graphext to write back the project's output to Airtable. Select “Allow write output.” View more documentation about this step in our [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# Amazon S3
Source: https://docs.graphext.com/documentation/import-and-export/data-connections/amazon-s3
Connect Graphext to Amazon S3 and be able to bring your data and write back recurrently
### Guided steps
Click on “New Project” (this will add the integration to your
personal team) or the “Integrations” button inside the team you want
to work on.
Click on **Security Credentials**, which you will find in the
dropdown in the top right corner.
Copy the **Access Key ID**, and paste it into the corresponding
field.
Copy your **Secret Access Key** (in case you don’t find it; below we
explain how to generate one), and paste it into the field labeled
**AWS Secret**.
Use the **Region name** dropdown to select the AWS region name of
your bucket.
Copy the **Bucket Name** in the Buckets tab and paste it into the
corresponding field.
Enter the desired bucket, and copy the **file path** you want to
connect to. If you have folders inside the bucket, follow this
format *“folder/example\_dataset.csv”*. Paste it into the **Path**
field.
If you want Graphext to write back the project's output to Amazon
S3. Select “Allow write output.” View more documentation about this
step in our [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin
exploring or “Save” the integration for later use.
### Create an Access Key
# Redshift
Source: https://docs.graphext.com/documentation/import-and-export/data-connections/aws-redshift
Connect Graphext to Amazon Redshift and be able to bring your data and write back to your data warehouse recurrently
(Make sure that your cluster is available for connection and that you have configure the IP correctly, in case you need help, we explain you how to do it in the section below "Configure the availability of your cluster and the IP")
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
select the cluster you want to use.
Copy the **Database name**, the **Port**, and the **Host** and paste them into the corresponding fields.
Using the AWS console or your favorite editor, and paste it in the field labeled **Query**. Make sure to access the full path.
If you want Graphext to write back the project's output to Amazon Redshift. Select “Allow write output.” View more documentation about this step in our [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
### Configure the availability of your cluster and the IP
Ensure you configure the Inbound rules with the correct IP and Port.
# Azure Blob Storage
Source: https://docs.graphext.com/documentation/import-and-export/data-connections/azure-blob
Connect Graphext to Azure Blob Storage and be able to bring your data and write back recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Head to Azure Blob Storage webpage, and select the desired **Storage account**.
In the **Security + networking** tab, click on the **Access Keys**. Copy the **Storage Account** name, and paste it into the field labeled **Account Name**.
Copy the **Key** from Azure, and paste it into the field labeled **Account Key**.
Navigate to the **Containers tab** under the Data Storage section, copy the name of the desired container, and paste it into the corresponding field.
Enter the desired container, copy the file name, and paste it into the field labeled **Blob Name**.
If you want Graphext to write back the project's output to Big Query. Select “Allow write output.” View more documentation about this step in [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# Azure SQL
Source: https://docs.graphext.com/documentation/import-and-export/data-connections/azuresql
Connect Graphext to a Azure SQL and be able to bring your data and write back recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Complete the form by entering your **"Username"**, **"Password"**, **"Host"**, **"Port"** and **"Database key"**.
Using your favorite editor, and paste it in the field labeled **Query**.
If you want Graphext to write back the project's output to Azure SQL. Select “Allow write output.” View more documentation about this step in our [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# Overview
Source: https://docs.graphext.com/documentation/import-and-export/data-connections/data-connections
Get your data straight from the source
If you are working with data stored in a database or hosted somewhere on the cloud, it's often simpler and more efficient to load your data directly into Graphext. By connecting a database or remotely hosted dataset with Graphext, you create a link between your workspace and the source of your data. You only have to integrate a data source with your Graphext workspace once.
Then each time you want to update your project with the latest data, you would [refresh the project](/documentation/import-and-export/update-data). This action will retrieve your data in its most up to date form, meaning that you can start analyzing recent data straight away.
## Available Data Connections
To read more about the connections that we currently support and how to configure them,
check out any of these references here.
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
## Allow connections from Graphext IP addresses
Numerous data sources are secured behind firewalls, necessitating Graphext's IP addresses to be permitted ('whitelisting') for connection. This typically involves modifying your firewall settings.
Please include the following IP addresses into the data source whitelist to allow Graphext to connect correctly.
```
34.32.182.228
34.34.16.28
34.140.25.149
34.140.138.132
```
# Databricks
Source: https://docs.graphext.com/documentation/import-and-export/data-connections/databricks
Connect Graphext to Databricks and be able to bring your data and write back to your data warehouse recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Go to SQL **Warehouses**, select the one you want to work with, and click on the **connection details** section
Copy the **Server hostname**, and paste it into the field labeled **Host**.
Copy the **HTTP path**, and paste it into the corresponding field.
Head to your Databricks **User Settings**, select the **Access Tokens** tab,and copy your token or generate a new one. Paste it into the appropriate field.
Using the Databricks SQL editor or your favorite editor, and paste it in the field labeled **Query**.
If you want Graphext to write back the project's output to Databricks. Select “Allow write output.” View more documentation about this step in our [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# Google BigQuery
Source: https://docs.graphext.com/documentation/import-and-export/data-connections/google-bigquery
Connect Graphext to Google BigQuery and be able to bring your data and write back to your data warehouse recurrently
### Guided steps
Click on “New Project”(this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Select your Google account and ”Allow” from the Google sign-in permissions window
Head to the Big Query webpage and copy your Project ID by clicking on the dropdown to the right of the Google Cloud logo. Paste it in the field labeled “Project ID.”
Create your SQL query using the Big Query editor or your favorite editor, and paste it in the field labeled “Query”.
If you want Graphext to write back the project's output to Big Query. Select “Allow write output.” View more documentation about this step in [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# Google Cloud Storage
Source: https://docs.graphext.com/documentation/import-and-export/data-connections/google-cs
Connect Graphext to Google Cloud Storage and be able to bring your data and write back recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Select your Google account and ”Allow” from the Google sign-in permissions window
Head to the Google Cloud webpage and open the **Buckets Section**. Copy the name of the desired bucket, and paste it into the field labeled **Bucket Name**.
Open the desired bucket in Google CS, and copy the file name you want to import. Paste it in the field labeled **Source Blob Name**.
Copy your Project ID by clicking on the dropdown to the right of the Google Cloud logo. Paste it in the field labeled **Project ID**.
If you want Graphext to write back the project's output to Big Query. Select “Allow write output.” View more documentation about this step in [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# Google Drive
Source: https://docs.graphext.com/documentation/import-and-export/data-connections/google-drive
Connect Graphext to Google Drive and be able to bring your data and write back recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Select your Google account and ”Allow” from the Google sign-in permissions window
Click on the three dots menu at the top right corner of the file and select **Get Link**.
Change the General access to Anyone with the link, copy it, and paste it into the field labeled **Shareable link**.
If you want Graphext to write back the project's output to Big Query. Select “Allow write output.” View more documentation about this step in [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# Google Sheets
Source: https://docs.graphext.com/documentation/import-and-export/data-connections/google-sheets
Connect Graphext to Google Sheets and be able to bring your data and write back recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Select your Google account and ”Allow” from the Google sign-in permissions window
Copy the URL from the browser (not the shareable link) and paste it into the field labeled **Browser URL of the desired sheet**. Note we can only retrieve the data from the sheet you select, not all the sheets in your file.
If you want Graphext to write back the project's output to Big Query. Select “Allow write output.” View more documentation about this step in [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# MySQL
Source: https://docs.graphext.com/documentation/import-and-export/data-connections/mysql
Connect Graphext to a MySQL and be able to bring your data and write back recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Complete the form by entering your **"Username"**, **"Password"**, **"Host"**, **"Port"** and **"Database key"**.
Using your favorite editor, and paste it in the field labeled **Query**.
If you want Graphext to write back the project's output to MySQL. Select “Allow write output.” View more documentation about this step in our [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# Notion
Source: https://docs.graphext.com/documentation/import-and-export/data-connections/notion
Connect Graphext to Notion and be able to bring your data and write back to your data warehouse recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Navigate and select the pages you want and click on the **Allow access** button.
Head to Notion, go to the three dots menu on top of the database you want to import, select **Link to view** and paste it into the field labeled **Link to the database view**
If you want Graphext to write back the project's output to Big Query. Select “Allow write output.” View more documentation about this step in [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# Data Sources Connections
Source: https://docs.graphext.com/documentation/import-and-export/data-connections/overview
Import and export from and to the most popular data sources
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
# PostgreSQL
Source: https://docs.graphext.com/documentation/import-and-export/data-connections/postgresql
Connect Graphext to a PostgreSQL and be able to bring your data and write back recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Complete the form by entering your **"Username"**, **"Password"**, **"Host"**, **"Port"** and **"Database key"**.
Using your favorite editor, and paste it in the field labeled **Query**.
If you want Graphext to write back the project's output to PostgreSQL. Select “Allow write output.” View more documentation about this step in our [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# Snowflake
Source: https://docs.graphext.com/documentation/import-and-export/data-connections/snowflake
Connect Graphext to Snowflake and be able to bring your data and write back to your data warehouse recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Open the **Admin** page and click on **Accounts**.
Click the icon to copy the URL. Paste it in the field labeled **Account ID / URL**.
Using the Snowflake Worksheets or your favorite editor, and paste it in the field labeled **Query**. Make sure to access the full path.
If you want Graphext to write back the project's output to Snowflake. Select “Allow write output.” View more documentation about this step in our [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# SQL Server
Source: https://docs.graphext.com/documentation/import-and-export/data-connections/sqlserver
Connect Graphext to a SQL Server and be able to bring your data and write back recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Complete the form by entering your **"Username"**, **"Password"**, **"Host"**, **"Port"** and **"Database key"**.
Using your favorite editor, and paste it in the field labeled **Query**.
If you want Graphext to write back the project's output to SQL Server. Select “Allow write output.” View more documentation about this step in our [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# Tinybird
Source: https://docs.graphext.com/documentation/import-and-export/data-connections/tinybird
Connect Graphext to Tinybird and be able to bring your data and write back to your data warehouse recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Head to the Tinybird webpage and click the **Auth tokens** section. Copy your token and paste it into the field labeled **Access Token**.
Copy the name of the desired pipe, and paste it into the field labeled **Pipe**. Make sure that the pipe is connected to a **data source**.
If you want Graphext to write back the project's output to Snowflake. Select “Allow write output.” View more documentation about this step in our [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# URL Endpoint
Source: https://docs.graphext.com/documentation/import-and-export/data-connections/url-endpoint
Fetch a specified endpoint of your choice to bring your data from anywhere
Click on “New Project” (this will add the integration to your personal team)
or the “Integrations” button inside the team you want to work on.
You may have to scroll down a bit to be able to see it.
![title]()
Give the project a name of your choice and paste the URL to your data in the "Public Link" field.
![title]()
The Method and Headers parameters will usually work fine by default.
In case your endpoint needs special headers for things like authentication tokens, you can add them in the field below. You can add more headers by clicking "ADD NEW ROW" below the "key" field.
Click "SAVE". Go into your project to see your data coming through!
# Enrichments and external providers
Source: https://docs.graphext.com/documentation/import-and-export/data-enrichments
API Keys, APIFY, Phantombuster, Dataset examples, Kaggle, etc
# Export your data
Source: https://docs.graphext.com/documentation/import-and-export/export-data
Bring your data with you anywhere
Graphext allows you to export your data in several ways, either directly downloading it or even
exporting it to your integration of choice.
## Downloading your data
The most straightforward way to export your data is to simply donwload it.
![Download Menu]()
You can download the data in 3 different ways:
1. Download the **original dataset**, without modifications
2. Download a subsection of the dataset as an **Excel®** file
3. Download a subsection of the dataset as a **CSV** file
For the last two options, this menu will pop up.
![Download Popup]()
This will allow to make a selection of the columns you want to download.
As stated in the blue note, any selection you have currently applied in your data will
be used to filter the resulting download.
In the case you created a graph or a model, the worflows create some utility
columns as calculations from your data. These are excluded from the selection
by default, but can be added back if desired.
## Exporting to an integration
In the case you have an [integration](/integrations/data-sources/overview) setup and **give it write access**, you can export any data to it.
For this, you have to setup **at least one integration** in which the option "Allow Write Ouput" is enabled. This
will allow Graphext to dump any data onto it.
![Allow Write Output]()
Projects and integrations are not necessarily related. You can export data
from any project to any integration as long as the given integration had the
option "Allow Write Ouput" when it was created.
If you (or someone in your team) enabled an integration with write access, this menu will become available:
![Export to integration option]()
If the project you are trying to export from belongs to a team, and no one in the team created an integration,
this message will show up:
![Export to integration disabled]()
Any member of the team can create an integration that any member of the team will be able to use once setup.
If everything is setup correctly, clicking the "Export to integration" button will take you to same sampling data
menu previously discussed. Select the columns (and apply the filters in the project) that you desire and click "Next".
This next menu will ask for the integration to which the data will be exported.
![Export menu]()
The integrations setup can be seen in the left sidebar. Here *every* integration is displayed, but only
those with write access can be chosen.
You can then save and export your data to said integration. The data will be exported attending to the
integration's format.
# Import data files
Source: https://docs.graphext.com/documentation/import-and-export/import-files
## Uploading files
In this section, we'll guide you through the process of uploading files to Graphext, a key step in beginning your project. Our platform supports a wide array of file types, ensuring versatility and compatibility with your data needs. Below, you'll find a detailed list of the [supported file formats](#file-formats).
Step-by-Step Guide to Uploading Your Files:
Begin by clicking on "Create a New Project". You can do this within your
personal team space or in any of your collaborative teams.
* **Browse**: Navigate through your computer's directories to find the
dataset you wish to use.
* **Drag and Drop:** Alternatively, you can drag
your file and drop it into the designated importation box on our platform.
Once you've selected your file, our system will upload the data and
automatically, infer the data types if needed and create a new project
containing your file. This process ensures your data is ready for
exploration and analysis.
After the upload is complete, your project is ready. You can now open it and
start your exploratory journey with Graphext.
*Additional Notes*:
* **Uploading Multiple Files with the Same Schema**: If you have several files with the same schema, there's no need to upload them one by one. Simply compress them into a ZIP file and upload it. Our system will seamlessly combine these files into one dataset for you.
* **Need Help with Complex Data Joins?**: For more intricate requirements, such as specific joins of different datasets, don't hesitate to reach out. Our team is on standby to assist you with any preprocessing needs. [Contact us](mailto:support@graphext.com).
## Supported File Types
In most cases, Graphext will inspect the raw data to try and infer the correct data type for each column
(categorical, numeric, date, etc).
This is not the case for formats that already have well defined column types,
such as Apache Arrow (`.arr` / `.arrow`), Parquet (`.pqt` / `.parquet`), and SPSS (`.sav`).
In these cases, instead of inferring the data types, we simply map them to the Graphext equivalent.
### File Formats
Extensions: .csv and .tsv
[CSV](https://en.wikipedia.org/wiki/Comma-separated_values) files (comma-separated values) are delimited text files using commas to separate values. Each line of the file contains a data record, and each record contains one or more fields separated by commas. Graphext expects column names to be listed in the first line of the file. TSV files (.tsv) follow the same formatting rules as CSV files but use the tab character ("\t") to delimit fields.
Graphext treats CSV and TSV as equivalent, and will try to infer the delimiter and other details of the format from the file's content. Note that if the file is not formatted correctly, or uses unusual characters as delimiters, or to quote fields containing the delimiter, Graphext may fail to infer the correct format and consequently to read the file correctly. In particular, while Graphext will try to skip initial lines that don't appear to be part of actual tabular data, this cannot be guaranteed to always work correctly, and so we discourage use of such preambles in CSV files.
For the curious, CSV files are read using Graphext's open-source [lector](https://github.com/graphext/lector) library, which is documented in some detail [here](https://lector.readthedocs.io/en/latest/).
Extensions: .xls and .xlsx
We support both XLS and XLSX files, the most commonly used formats to save Microsoft Excel spreadsheets. These files store data in worksheets containing cells arranged as a grid of rows and columns. Like other file types you can upload these directly to Graphext. In case the file has several sheets, we will import the **first sheet only**.
To ensure Graphext reads your data correctly, we recommend that the sheet contain a single table only, that the table start in the first row and first column, and that the first row correspond to the table's column names. If the sheet contains comments, charts or other elements not part of the tabular data, the import may not work as expected.
Extensions: .json and .jsonl
[JSON](https://en.wikipedia.org/wiki/JSON) files ("JavaScript Object Notation") are primarily used for transmitting data between web applications and servers. They store data in a format similar to a JavaScript object or Python dictionary. You can upload JSON files directly to Graphext. We support normal JSON files (.json) as well as [line-delimited json](https://jsonlines.org/) files (.jsonl). See below for details about the supported file structures (both row- and column-oriented).
Note that we currently do not support the import of nested data. Any column containing nested JSON data will be imported as plain strings. You will nevertheless be able to extract specific fields from nested data using our step "*extract\_json\_values*" find this step in our [API Docs](/api-docs)
Extensions: .arr and .arrow
You can also upload binary [Apache Arrow](https://arrow.apache.org/) files written in [streaming or random access](https://arrow.apache.org/docs/python/ipc.html) (batch) mode to Graphext. Arrow represents a language-independent columnar memory format for flat and hierarchical data. Using this format allow for very fast imports, since the format is very efficient to read, compact, and doesn't require inference of data types. It is the format Graphext and many other tools, DBs etc. use internally to store their data.
As with JSON, we currently do not support the import of nested data. Any column containing nested data will be imported as plain strings. You will nevertheless be able to extract specific fields from nested data using one of our data transformations steps after import ([API Docs](/api-docs)). Other types not currently supported will either be imported as categorical (text), or be represented by a column containing only missing values (to at least preserve the correct number of columns).
Extensions: .pqt and .parquet
[Apache Parquet](https://parquet.apache.org/) is a widely used open source, column-oriented data file format designed for efficient data storage and retrieval. Importing it into Graphext is equivalent to importing Apache Arrow files, offering the same performance benefits (and with same caveats regarding unsupported data types).
Extension: .sav
SAV files are part of the SPSS Statistics File Format or SPSS family. Information in a SAV file is divided into a header, a sequence of tagged 'records' comprising a dictionary for the file and finally the data itself.
Note that while we try our best to import such files, SAV is a proprietary format with no official documentation, and as such is not well supported in the greater data ecosystem. If you are able to export your data in another of our supported formats we would recommend that instead.
Extensions: .gml and .graphml
The file formats [Graph Modelling Language](https://en.wikipedia.org/wiki/Graph_Modelling_Language) (.gml), and [GraphML](https://en.wikipedia.org/wiki/GraphML) (.graphml), let you import data already representing a graph or network. They can be exported from tools like [Gephi](https://gephi.org/) or [Cytoscape](https://cytoscape.org/), or libraries such as [igraph](https://igraph.org/) or [NetworkX](https://networkx.org/). We will be able to read those files as long as igraph is able to read them. This means some advanced features, for example hypergraphs or ports, are not supported.
Extension: .sav
Zip files allow you to upload and concatenate multiple dataset files at once. The archive should contain files in any of the supported formats mentioned, and all files should share the same schema (column names and data types). Graphext will concatenate all contained datasets horizontally, i.e. by appending their rows, and so the columns must best consistent across different files.
Additionally, Graphext will automatically detect and convert the following list of strings to missing values (equivalent to no value, or and empty cell):
```
"#N/A", "#N/A N/A", "#NA", "-1.#IND", "-1.#INF", "-1.#QNAN", "-NaN",
"-nan", "1.#IND", "1.#INF", "1.#INF000000", "1.#QNAN", "", "N/A",
"n/a", "NA", "NAN", "NaN", "nan", "NULL", "Null", "null", ""
```
The conversion will apply only if the whole field corresponds to one of these strings, i.e. if any of these values occurs as a substring inside a longer text, it will be left unchanged.
### Correct File Structures
Text-like file formats, like CSV and JSON, may be subject to specific restrictions on how the data is structured inside the file.
While there is no "official" **CSV standard**, most implementations follow some **common rules**. We recommend adhering to the **following guidelines** adapted from the [Internet Engineering Task Force](https://en.wikipedia.org/wiki/Internet_Engineering_Task_Force), which you may also access directly [here](https://datatracker.ietf.org/doc/html/rfc4180#page-2).
The first line in the file is a header line with the same format as normal record lines. This header contains names corresponding to the fields in the file and should contain the same number of fields as the records in the rest of the file. For example:
```
field_1,field_2,field_3
aaa,bbb,ccc
zzz,yyy,xxx
```
Each actual data record is located on a **new line**, delimited by a line break.
The **last record** in the file may or may not have an **ending line break**.
Within the header and each record, there may be one or more fields, separated by commas. Each line should contain the same number of fields throughout the file. Spaces are considered part of a field and will not be ignored. The **last field in the record must not be followed by a comma**. For example:
✅ **GOOD**
```
field_1,field_2,field_3
aaa,bbb,ccc
zzz,yyy,xxx
```
❌ **BAD**
```
field_1,field_2,field_3
aaa,bbb,ccc,
zzz,yyy,xxx,
```
Each field may or may not be **enclosed in double quotes**. If fields are not enclosed with double quotes, then double quotes may not appear inside the fields. For example:
```
"aaa","bbb","ccc"
zzz,yyy,xxx
```
Fields containing **line breaks, double quotes, and commas must be enclosed in double-quotes**. For example:
```
"aaa","b
bb","ccc"
zzz,yyy,xxx
```
If double-quotes are used to enclose fields, then a **double-quote appearing inside a field** must be **escaped** by preceding it with another double quote. For example:
```
"aaa","He said ""Hi!""","ccc"
```
We support **three different JSON (JavaScript Object Notation)** formats, which will be detected automatically by inspecting the beginning of a .json file.
**Json Lines:**
In the JSON lines format, each line in the file is a JSON object representing a dataset row. The object in each row contains field names as keys and the corresponding field's value. For example:
```JSON theme={null}
{"field_1": "aaa", "field_2": "bbb", "field_3": "ccc"}
{"field_1": "zzz", "field_2": "yyy", "field_3": "xxx"}
```
For further details see the official [JSON Lines documentation](https://jsonlines.org/).
**List of Records:**
In this format the file contains a JSON list of objects, where each object contains field names and values as **key-value pairs**. For example:
```JSON theme={null}
[
{"field_1": "aaa", "field_2": "bbb", "field_3": "ccc"},
{"field_1": "zzz", "field_2": "yyy", "field_3": "xxx"}
]
```
Notice how the first level represents a list, and that objects within this list are separated by a comma. Line breaks and spaces between fields are not required, so the following is an equivalent but more compact format that is equally valid:
```JSON theme={null}
[{"field_1":"aaa","field_2":"bbb","field_3":"ccc"},{"field_1":"zzz","field_2":"yyy","field_3":"xxx"}]
```
**Object of Columns:**
The last supported JSON format is column-oriented. In this format the file contains at the highest level a JSON object. This object has key-value pairs where each key is the name of a field/column, and each value is a JSON list containing `{index: value}` objects. For example:
```
{
"field_1": {0: "aaa", 1: "zzz"},
"field_2": {0: "bbb", 1: "yyy"},
"field_3": {0: "ccc", 1: "xxx"}
}
```
In this format, line breaks and spaces between fields are also ignored, and so the following is equivalent:
```
{"field_1":{0:"aaa",1:"zzz"},"field_2":{0:"bbb",1:"yyy"},"field_3":{0:"ccc",1:"xxx"}}
```
***A Note on Automatic Detection***
As can be seen in the examples, each JSON format is easily identified by inspecting the first few lines of the file. We use the following heuristic:
1. If the file starts with `[` - assume the List of Records format.
2. If the file contains more than 1 line, and each of the first 2 lines starts with `{` and ends with `}` - assume the JSON Lines format.
3. In all other cases - assume the Object of Columns format.
# Update your data recurrently
Source: https://docs.graphext.com/documentation/import-and-export/update-data
Setting up Graphext to update your data on a periodic basis
When dealing with data that comes from a given [integration](/integrations/data-sources/overview),
you can tell Graphext to update it recurrently, so you can
have the latest version always available and ready to be analyzed.
Doing this is very easy:
[Set your project up](/documentation/import-and-export/data-connections) as usual, connecting your preferred integration.
On the top right corner there should be a little cloud icon. Click on it to open the dropdown. At the very bottom, you can see **Recurrence: Not scheduled**. Click and continue.
![Recurrence]()
This menu will pop up, prompting you for some settings.
![Recurrence menu]()
You can set up:
* **Type of recurrence**
* **Refresh**: will overwrite your last dataset with the new one
* **Refresh & Backup**: will save each version of your data as a snapshot
* **Recurrence pattern**: Define how much time will span in between updates
When configuring a weekly, monthly or anual pattern, the beginning of the period
starts at:
* Weekly: every Monday
* Monthly: every first day of the following month
* Anually: every first day of the following year
For hourly, daily and weekdays, the pattern just depends on the time. Weekdays will update the data
every day except for Saturdays and Sundays.
* **Time Zone**: you may select a different time zone if that is of your convenience
* **Time**: exactly at what point in the period should the data be updated
The **Next run** row will display when the pattern was first created, when was the last
time it was updated, and when is the next update scheduled. This information will be
populated upon saving your settings.
This won't work on data that comes from a file from your computer since we
need to periodically bring the latest data from the source.
# API Keys Connections
Source: https://docs.graphext.com/documentation/integrations/api-keys/overview
Enrinch your data with AI, bring data from social media or use your favourite ML model
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
# Airtable
Source: https://docs.graphext.com/documentation/integrations/data-sources/airtable
Connect Graphext to Airtable and be able to bring your data and write back to your data warehouse recurrently
![]()
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Head to your Airtable and open the dataset you want to import. From the URL you need to extract the **Base ID** and the **Table ID**
* The Table ID is the first part of the URL and starts with "app..."
* The Base ID id the second part of the URL and starts with "tbl..."
![]()
To obtain the Access Token you need to navigate to [https://airtable.com/create/tokens](https://airtable.com/create/tokens). Click on "Create new token", define the name the scope and the access. At least you should include the two scopes to read (data.records:read and schema.bases:read) and give access to the table you want to import
![]()
Copy the generated token
![]()
Finally you need to include the **Table ID** the **Base ID** and the **Access Token** in the Graphext configuration UI
![]()
If you want Graphext to write back the project's output to Airtable. Select “Allow write output.” View more documentation about this step in our [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# Amazon S3
Source: https://docs.graphext.com/documentation/integrations/data-sources/amazon-s3
Connect Graphext to Amazon S3 and be able to bring your data and write back recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Click on **Security Credentials**, which you will find in the dropdown in the top right corner.
Copy the **Access Key ID**, and paste it into the corresponding field.
Copy your **Secret Access Key** (in case you don’t find it; below we explain how to generate one), and paste it into the field labeled **AWS Secret**.
Use the **Region name** dropdown to select the AWS region name of your bucket.
Copy the **Bucket Name** in the Buckets tab and paste it into the corresponding field.
Enter the desired bucket, and copy the **file path** you want to connect to. If you have folders inside the bucket, follow this format *“folder/example\_dataset.csv”*. Paste it into the **Path** field.
If you want Graphext to write back the project's output to Amazon S3. Select “Allow write output.” View more documentation about this step in our [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
### Create an Access Key
# Redshift
Source: https://docs.graphext.com/documentation/integrations/data-sources/aws-redshift
Connect Graphext to Amazon Redshift and be able to bring your data and write back to your data warehouse recurrently
(Make sure that your cluster is available for connection and that you have configure the IP correctly, in case you need help, we explain you how to do it in the section below "Configure the availability of your cluster and the IP")
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
select the cluster you want to use.
Copy the **Database name**, the **Port**, and the **Host** and paste them into the corresponding fields.
Using the AWS console or your favorite editor, and paste it in the field labeled **Query**. Make sure to access the full path.
If you want Graphext to write back the project's output to Amazon Redshift. Select “Allow write output.” View more documentation about this step in our [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
### Configure the availability of your cluster and the IP
Ensure you configure the Inbound rules with the correct IP and Port.
# Azure Blob Storage
Source: https://docs.graphext.com/documentation/integrations/data-sources/azure-blob
Connect Graphext to Azure Blob Storage and be able to bring your data and write back recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Head to Azure Blob Storage webpage, and select the desired **Storage account**.
In the **Security + networking** tab, click on the **Access Keys**. Copy the **Storage Account** name, and paste it into the field labeled **Account Name**.
Copy the **Key** from Azure, and paste it into the field labeled **Account Key**.
Navigate to the **Containers tab** under the Data Storage section, copy the name of the desired container, and paste it into the corresponding field.
Enter the desired container, copy the file name, and paste it into the field labeled **Blob Name**.
If you want Graphext to write back the project's output to Big Query. Select “Allow write output.” View more documentation about this step in [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# Azure SQL
Source: https://docs.graphext.com/documentation/integrations/data-sources/azuresql
Connect Graphext to a Azure SQL and be able to bring your data and write back recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Complete the form by entering your **"Username"**, **"Password"**, **"Host"**, **"Port"** and **"Database key"**.
Using your favorite editor, and paste it in the field labeled **Query**.
If you want Graphext to write back the project's output to Azure SQL. Select “Allow write output.” View more documentation about this step in our [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# Databricks
Source: https://docs.graphext.com/documentation/integrations/data-sources/databricks
Connect Graphext to Databricks and be able to bring your data and write back to your data warehouse recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Go to SQL **Warehouses**, select the one you want to work with, and click on the **connection details** section
Copy the **Server hostname**, and paste it into the field labeled **Host**.
Copy the **HTTP path**, and paste it into the corresponding field.
Head to your Databricks **User Settings**, select the **Access Tokens** tab,and copy your token or generate a new one. Paste it into the appropriate field.
Using the Databricks SQL editor or your favorite editor, and paste it in the field labeled **Query**.
If you want Graphext to write back the project's output to Databricks. Select “Allow write output.” View more documentation about this step in our [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# Google BigQuery
Source: https://docs.graphext.com/documentation/integrations/data-sources/google-bigquery
Connect Graphext to Google BigQuery and be able to bring your data and write back to your data warehouse recurrently
### Guided steps
Click on “New Project”(this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Select your Google account and ”Allow” from the Google sign-in permissions window
Head to the Big Query webpage and copy your Project ID by clicking on the dropdown to the right of the Google Cloud logo. Paste it in the field labeled “Project ID.”
Create your SQL query using the Big Query editor or your favorite editor, and paste it in the field labeled “Query”.
If you want Graphext to write back the project's output to Big Query. Select “Allow write output.” View more documentation about this step in [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# Google Cloud Storage
Source: https://docs.graphext.com/documentation/integrations/data-sources/google-cs
Connect Graphext to Google Cloud Storage and be able to bring your data and write back recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Select your Google account and ”Allow” from the Google sign-in permissions window
Head to the Google Cloud webpage and open the **Buckets Section**. Copy the name of the desired bucket, and paste it into the field labeled **Bucket Name**.
Open the desired bucket in Google CS, and copy the file name you want to import. Paste it in the field labeled **Source Blob Name**.
Copy your Project ID by clicking on the dropdown to the right of the Google Cloud logo. Paste it in the field labeled **Project ID**.
If you want Graphext to write back the project's output to Big Query. Select “Allow write output.” View more documentation about this step in [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# Google Drive
Source: https://docs.graphext.com/documentation/integrations/data-sources/google-drive
Connect Graphext to Google Drive and be able to bring your data and write back recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Select your Google account and ”Allow” from the Google sign-in permissions window
Click on the three dots menu at the top right corner of the file and select **Get Link**.
Change the General access to Anyone with the link, copy it, and paste it into the field labeled **Shareable link**.
If you want Graphext to write back the project's output to Big Query. Select “Allow write output.” View more documentation about this step in [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# Google Sheets
Source: https://docs.graphext.com/documentation/integrations/data-sources/google-sheets
Connect Graphext to Google Sheets and be able to bring your data and write back recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Select your Google account and ”Allow” from the Google sign-in permissions window
Copy the URL from the browser (not the shareable link) and paste it into the field labeled **Browser URL of the desired sheet**. Note we can only retrieve the data from the sheet you select, not all the sheets in your file.
If you want Graphext to write back the project's output to Big Query. Select “Allow write output.” View more documentation about this step in [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# MySQL
Source: https://docs.graphext.com/documentation/integrations/data-sources/mysql
Connect Graphext to a MySQL and be able to bring your data and write back recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Complete the form by entering your **"Username"**, **"Password"**, **"Host"**, **"Port"** and **"Database key"**.
Using your favorite editor, and paste it in the field labeled **Query**.
If you want Graphext to write back the project's output to MySQL. Select “Allow write output.” View more documentation about this step in our [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# Notion
Source: https://docs.graphext.com/documentation/integrations/data-sources/notion
Connect Graphext to Notion and be able to bring your data and write back to your data warehouse recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Navigate and select the pages you want and click on the **Allow access** button.
Head to Notion, go to the three dots menu on top of the database you want to import, select **Link to view** and paste it into the field labeled **Link to the database view**
If you want Graphext to write back the project's output to Big Query. Select “Allow write output.” View more documentation about this step in [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# OpenAI
Source: https://docs.graphext.com/documentation/integrations/data-sources/open-ai
Enhance your projects with the latest OpenAI models
This section explains how to obtain an api key from openAI and configure it in a
Graphext team. This allows you to use the OpenAI API in your Graphext projects,
with most notable steps like
[prompt\_ai](/api-docs/analyse/infer/prompt_ai#prompt-ai), which allow for
inferring knowledge and enriching datasets using LLMs.
## Steps
There are two steps to this process:
1. [Get the key from openAI](/documentation/integrations/data-sources/open-ai#get-the-key-from-openai)
2. [Use the key in Graphext](/documentation/integrations/data-sources/open-ai#use-the-key-in-graphext)
## Get the key from openAI
Navigate to [OpenAI - API Keys](https://platform.openai.com/account/api-keys) and generate
a new key. Depending on your permissions, you may ask your manager/employer to
enable them for you.
This should open a screen like this:
![Open AI Platform]()
Towards the top right corner you'll see a "Create Secret Key" button. Click it,
and you'll be asked for a name. This name can be anything you want. It's purpose
it's to identify it uniquely, helping you know which keys are spending most.
Once you've created the key, you'll be able to copy it and use it in Graphext.
Once the key is created, you will not be able to see it again. Make sure to
save it in a secure place, like a password manager.
If you lose it, **no worries**. Just delete it from the panel and create a new one.
## Use the key in Graphext
Within Graphext, OpenAI keys are team-scoped. This means that if you or anyone in a team sets
up an OpenAI key, all members of the team will be able to use it.
This also means that for new teams, a key will have to be setup.
To use the key in Graphext, follow these steps:
Go to any team you want to enable the openAI integration on.
![Navigate team]()
Click on the "ADD INTEGRATION" button, towards the top center of the page.
If you already have integrations setup, this
button will display how many you have enabled.
![Add integration]()
Towards the top left corner, you'll see an "API Keys" section.
![Add integration]()
Towards the top left corner you'll see an "Add API Key" button.
![Add integration]()
Click on the "Open AI" icon to input your openAI key.
![Add integration]()
Input the key in the "Open AI Key" field.
The "**name**" field compulsory. It serves as a unique identifier to use in
your Graphext projects, like in the
[Recipe](/concepts/graphext-concepts/recipe) or the [Wizard](/concepts/graphext-concepts/wizard).
A good rule of thumb is to name it after whatever you're going to use it
for.
In this case, since it's going to help me analyze some tweets, I'll name it
gpt-tweets. I could name it whatever I want, like tweets-analyzer, or
tweets.
Keep in mind that there cannot be two integrations within the same team
that have the same name.
![Add integration]()
Like that, you can now use the immensely powerful capabilities of LLMs in
Graphext!
# Import from Integrations
Source: https://docs.graphext.com/documentation/integrations/data-sources/overview
Import and export from and to the most popular data sources
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
***
}
/>
# PostgreSQL
Source: https://docs.graphext.com/documentation/integrations/data-sources/postgresql
Connect Graphext to a PostgreSQL and be able to bring your data and write back recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Complete the form by entering your **"Username"**, **"Password"**, **"Host"**, **"Port"** and **"Database key"**.
Using your favorite editor, and paste it in the field labeled **Query**.
If you want Graphext to write back the project's output to PostgreSQL. Select “Allow write output.” View more documentation about this step in our [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# Snowflake
Source: https://docs.graphext.com/documentation/integrations/data-sources/snowflake
Connect Graphext to Snowflake and be able to bring your data and write back to your data warehouse recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Open the **Admin** page and click on **Accounts**.
Click the icon to copy the URL. Paste it in the field labeled **Account ID / URL**.
Using the Snowflake Worksheets or your favorite editor, and paste it in the field labeled **Query**. Make sure to access the full path.
If you want Graphext to write back the project's output to Snowflake. Select “Allow write output.” View more documentation about this step in our [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# SQL Server
Source: https://docs.graphext.com/documentation/integrations/data-sources/sqlserver
Connect Graphext to a SQL Server and be able to bring your data and write back recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Complete the form by entering your **"Username"**, **"Password"**, **"Host"**, **"Port"** and **"Database key"**.
Using your favorite editor, and paste it in the field labeled **Query**.
If you want Graphext to write back the project's output to SQL Server. Select “Allow write output.” View more documentation about this step in our [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# Tinybird
Source: https://docs.graphext.com/documentation/integrations/data-sources/tinybird
Connect Graphext to Tinybird and be able to bring your data and write back to your data warehouse recurrently
### Guided steps
Click on “New Project” (this will add the integration to your personal team) or the “Integrations” button inside the team you want to work on.
Head to the Tinybird webpage and click the **Auth tokens** section. Copy your token and paste it into the field labeled **Access Token**.
Copy the name of the desired pipe, and paste it into the field labeled **Pipe**. Make sure that the pipe is connected to a **data source**.
If you want Graphext to write back the project's output to Tinybird. Select “Allow write output.” View more documentation about this step in our [API Docs](/api-docs).
Click on “Save and create project” to retrieve your data and begin exploring or “Save” the integration for later use.
# URL Endpoint
Source: https://docs.graphext.com/documentation/integrations/data-sources/url-endpoint
Fetch a specified endpoint of your choice to bring your data from anywhere
Click on “New Project” (this will add the integration to your personal team)
or the “Integrations” button inside the team you want to work on.
You may have to scroll down a bit to be able to see it.
![title]()
Give the project a name of your choice and paste the URL to your data in the "Public Link" field.
![title]()
The Method and Headers parameters will usually work fine by default.
In case your endpoint needs special headers for things like authentication tokens, you can add them in the field below. You can add more headers by clicking "ADD NEW ROW" below the "key" field.
Click "SAVE". Go into your project to see your data coming through!
# Clustering
Source: https://docs.graphext.com/documentation/machine-learning/clustering
A modern visual advanced analytics platform for your business
# Other analysis
Source: https://docs.graphext.com/documentation/machine-learning/other-analysis
A modern visual advanced analytics platform for your business
# Predictive models
Source: https://docs.graphext.com/documentation/machine-learning/predictive-models
A modern visual advanced analytics platform for your business
# Text analysis
Source: https://docs.graphext.com/documentation/machine-learning/text-analysis
A modern visual advanced analytics platform for your business
# Account Settings
Source: https://docs.graphext.com/documentation/manage-workspace/account-settings
Manage your account preferences and much more
![Interface Overview]()
Account Overview is the place where you'll manage the information associated with
your account, and its preferences, configure the plan you are subscribed to, and
sign out.
In this documentation you'll learn to:
1. How to view and edit your **personal information**
* Change your name
* Change your profile picture
* Change the account password
2. How to customize your **theme**
3. How to manage your **subscription plan**
* How to add, remove, or edit seats in your subscription plan
4. How to configure **notifications**
5. How to **sign out**
## How to view and edit your personal info
This is where you can edit your username, change your profile picture, and reset your account password.
Click the **Account** dropdown menu in the navigation bar
![Interface Overview]()
Open **Account Settings > Manage Account**
![Interface Overview]()
Click on the **Change Name** button
![Interface Overview]()
Edit it in the **dialogue**
![Interface Overview]()
Apply changes. A banner will notify you that the changes have been
applied.
COMING SOON
Click the **Account** dropdown menu in the navigation bar
![Change Password]()
Open **Account Settings > Manage Account**
![Change Password]()
Click on the **Reset password** button. An email will be sent to the
email associated with your account.
![Change Password]()
Open the email and click on the **Reset Password** button or copy and
paste the URL link in the browser
Enter your new password in both inputs and click on the **Reset
Password** button. Make sure you follow the requirements.
## How to customize your theme
In Graphext, we understand the importance of a personalized user experience, which is why we offer two distinct themes: **dark and light**. You can switch between these themes at any point to suit your visual preference or lighting conditions. For your convenience, you can easily change the theme using the toggle available in the dropdown menu at the top left of the interface, or by adjusting your preferences in the account settings. Additionally, for those who desire an even more seamless experience, we provide an option to **synchronize the theme with your system settings**. This feature allows the Graphext interface to automatically adapt to the time of day, aligning with your system's theme to ensure optimal visibility and comfort throughout your day. Whether you prefer the stark elegance of dark mode or the crisp brightness of light mode, our customizable settings are designed to meet the needs of all users.
* Themes toggle
![title]()
* Account settings themes configuration
![title]()
## How to manage your subscription plan
Click the **Account** dropdown menu in the navigation bar
![Interface Overview]()
Open **Account Settings > Manage Account**
![Interface Overview]()
Click on the **Plans** tab
![Interface Overview]()
Select the **PRO** plan and proceed to checkout or [contact our sales team](mailto:support@graphext.com) to contract an **enterprise plan**
Click the **Account** dropdown menu in the navigation bar
![Interface Overview]()
Open **Account Settings > Manage Account**
![Interface Overview]()
Click on the **Members** tab
![Interface Overview]()
Click on the **+ Add Team Member** button and write the email you want to add to your team.
![Interface Overview]()
Click on the **permissions dropdown** to change the role or remove that account.
![Interface Overview]()
## How to configure notifications
We have designed our notification system to be minimally invasive, ensuring that it only alerts you to matters of genuine importance. This approach helps prevent the accumulation of unread notifications, a common issue in many software applications where excessive, less useful alerts can lead users to ignore them altogether. Key notifications you will receive include:
* Alerts when a project **execution has failed**
* Notifications when someone or something has **stopped a recurrence**
* Invitations to join a **new team**
* Updates when a **project has been shared** with you.
If you miss any other notification or you want to do a more advance configuration of them, please [contact us](mailto:support@graphext.com).
## How to sign out
You will find this option in the **Account** dropdown menu, in the navigation bar.
![title]()
# Manage connections
Source: https://docs.graphext.com/documentation/manage-workspace/manage-connections
A modern visual advanced analytics platform for your business
# Manage projects
Source: https://docs.graphext.com/documentation/manage-workspace/manage-projects
Understand what projects are and how to handle them
A project, is the **fundamental unit for analysis**. Each project is **linked to a dataset** that can originate from various sources. Visually, projects are represented as cards within the hall. Clicking on one of these cards transports you to the analysis area. Typically, it is advisable to dedicate a separate project to each type of analysis, although combining different analyses in a single project can be beneficial under certain circumstances. For instance, you might perform clustering and then use the newly created cluster variable as a predictor in a subsequent predictive model.
Projects are **inherently collaborative**. You can share them with other accounts, allowing multiple users to edit and contribute simultaneously. For instances where editing is not required, projects can be shared with viewers who need access to view the analyses but not modify them.
Projects connected to a data source can **automatically update** when the originating data source is refreshed, ensuring that your analysis remains up to date.
Additionally, several actions can be performed on projects to manage and optimize your workflow effectively. These actions include **renaming, copying, moving projects** to different teams, **removing projects**, or even **starting anew from scratch**. The latter resets the project to its original state, preserving the initial dataset and removing any transformations that have been applied.
## How to create a new project
From the Home page, you can initiate a new project by clicking the prominent blue **"New Project" button**. Alternatively, within each team's interface, the first card displayed is the **"+ New Project" card**, which is also clickable and serves as another gateway to start a new project.
![New project]()
![New project]()
Upon selecting either of these options, a dialog window will appear prompting you
to choose the source of your data. We provide three primary methods for importing
data into your project:
* **Upload a file**: Directly upload data files from your local storage.
* **Connect to a data integration**: Utilize connections to a variety of data integrations available on our platform.
* **Use a data example**: Start with one of our pre-loaded data examples to get a feel for the system or to conduct test analyses.
For more detailed information on these options, please refer to the [Import Data section](/documentation/import-and-export).
Once you have successfully selected and imported a dataset, a new project card will appear in the hall. This card represents your new project, and clicking on it will take you directly to the analysis area, where you can begin exploring and interpreting your data.
## How to share a project with a team member and change role
Facilitates collaboration on the same analysis with numerous users. Depending on your specific needs, you can share projects as either editors or viewers. Editors are granted permissions to contribute to the project actively, while viewers are able to access the project solely for the purpose of consuming insights without making any alterations.
To share a project, simply follow these steps:
Navigate to the project card within the hall and click on the menu to find
the **"Share"** option.
![Sharing a project]()
A window will appear where you can search for and **enter the emails of
the users** with whom you want to share the project.
![Sharing a project]()
If at any point you need to adjust user **permissions or roles**, you can
easily manage these settings. Return to the same menu, select the "Share"
option again, and you will have the option to modify the roles of existing
users or remove them entirely.
![Sharing a project]()
## How to refresh a project and add a recurrence
A project linked to an integration, allows the dataset to be updated, with two methods to rerun the analysis on this new data: **manually or automatically**.
* For **manual updates**, you can utilize the **"Refresh Project"** feature. Simply click on the menu of the project card and select "Refresh Project." This action triggers the system to reapply the existing ["recipe"](/concepts/graphext-concepts/recipe) to the updated dataset, ensuring your analysis reflects the latest data.
![Refreshing projects]()
* For **automatic updates**, you can set up a **"Recurrence"** by clicking "Add Recurrence" in the project card menu. This feature automates the execution of the project at a frequency you specify, allowing for hands-off updates and analysis. This can be particularly useful for projects requiring regular data refreshes to maintain accuracy and relevancy over time. You can learn more about setting up and managing recurrences [here](/documentation/import-and-export/update-data). You can always **edit the frequency** or **pause the recurrence**.
![Adding recurrence]()
## How to search, filter and order projects.
### Omnibar
To locate any team or project, regardless of how extensive your list grows, you can use the **omnibar**.
To access the omninbar you have to options, click on the **search bar at the top** or by using **keyboard shortcuts**. If you are on a Mac, press **Cmd + K**; on Windows, use **Ctrl + K**. This omnibar searches across both teams and projects.
![Searching projects]()
### Filter and sorting If you are looking for a specific project, apart from using
the omnibar for a direct search, you can apply filters and sorting options to organize
your view more effectively.
* **Filter Projects**: Located at the top right corner, the filtering options allow you to quickly pinpoint projects based on specific criteria. This can include finding projects by their creator, those that use particular integrations, or ones created during a designated period
![Filtering projects]()
* **Sort Projects**: Also found at the top right corner, you can sort by the date of creation, project name, number of rows or other relevant metrics.
![Sorting projects]()
## How to pin a project
In order to pin a project, simply hover your mouse over its corresponding card. A thumbtack icon will
appear in the top left corner. Click it, and the project will be pinned to the top, always visible.
![Pin project]()
In case you want to un-pin a project, click on the same —now dashed— icon, and it'll go back to its original position.
![Pinned projects]()
## How to rename a project
To rename a project, just navigate to the project card menu and select the **"Rename"** option. Write the name and confirm the new one. Remember that you can use emojis 😉
![]()
## How to make a copy of a project
To make a copy of a project, simply go to the project card menu and select the **"Make a copy"** option. Choose the desired team from the dropdown menu and confirm your selection.
![Copying projects]()
## How to move a project to another team
In order to move a project to another team, you can click the three little blue dots on the project's card and then select "Move to...". This
will ask you to select the destination team of the project.
![Moving projects]()
When copying or moving projects, consider the following if said project is connected to an integration:
* **If you are an admin in both the originating and destination teams**, we
will also replicate the integration for the destination team. This newly
created integration will be independent from the original but will carry over
the same query and credentials. You can then manage it within the new team.
* **If you are not an admin in at least one of the teams**, we will only move
the project and not the integration, due to security reasons.
## How to remove a project
In order to delete a project, you can click the three little blue dots on the project's card and then select remove. Confirm if you
really want to delete the project before deleting.
![Deleting a project]()
Deleting a project is not reversible. Please, be careful and consider
thoughtfully before deleting.
# Manage teams
Source: https://docs.graphext.com/documentation/manage-workspace/manage-teams
Collaborate with your team mates and organize your projects in an efficient way
A team, is an organizational level designed to enhance collaboration among teammates. You can create **unlimited projects** within a team, **invite various users** to join, and assign them specific roles—**Viewer, Member, or Admin**—to tailor their access and capabilities according to their responsibilities. [Learn more about roles](/documentation/manage-workspace/manage-teams#type-of-roles).
Teams also come equipped with their own integrations, allowing members to leverage these tools according to their roles. Depending on their permissions, users can create new projects from these integrations, create new ones or edit existing onsecuritypolicyviolation. More details are available on our **Integrations page**.
The sidebar provides quick access to all teams you belong to. For ease of access, you can pin your most visited teams, ensuring they are always readily accessible. This page is divided into two main sections: 'Pinned Teams' for your frequently accessed teams and 'All Teams' for a comprehensive view of every team you belong to."
## How to create a new team
In the sidebar, where all your teams are listed, creating a new team is straightforward and quick. Simply **hover over the teams bar** and **click on the plus button** to get started. All you need to do is give your new team a name, and you're all set to invite colleagues, start new projects, and configure integrations to suit your team’s needs.
![Interface Overview]()
## Team members
### How to add or remove team members
Navigate to the desired team using the side panel. Once inside a team's
section, you'll find **key details on the top bar**, including the number
of team members, which provides a quick overview of your collaborative
environment.
![]()
Clicking on the team member icon will open a modal displaying the
information about the members of the team. Within this window, you can
search for and **input the email** of the user you wish to invite. During
this process, you will also **select the role** you want to assign to the
new member, tailoring their access and permissions within the team.
![]()
If you are an admin, in this same window, you will be able to change the
role or remove any member of the team
![]()
### Type of roles
In Graphext, we have established three distinct roles to accommodate various levels of access and control within teams. These roles ensure that team members can perform their tasks effectively while maintaining necessary security and organizational standards:
* **Admin**: As the most privileged role, Admins have comprehensive control over the team's dynamics. They can manage integrations, create new projects, and modify team settings, allowing them to tailor the team environment to suit specific needs.
* **Member**: Members have the ability to create projects, which enables active participation in ongoing tasks and initiatives. However, they do not have permission to manage integrations or alter team settings, maintaining a focus on project development without access to higher-level administrative functions.
* **Viewer**: Viewers have a more restricted role, designed primarily for oversight and monitoring. They can only view projects, making this role ideal for stakeholders or team members who need to stay informed about progress without directly altering project content or structure.
## How to pin and reposition a team
In Graphext, the functionality to pin teams is designed to streamline your workflow by making frequently visited teams more accessible. Pinning a team moves it to a dedicated section at the top of your interface, ensuring you can find and access these teams quickly, enhancing your efficiency. Here’s how you can pin a team:
Simply move your cursor over the team you want to pin. A menu icon will
appear on the right side of the team's name.
Click on the menu icon. A list of options will appear. From these, select
the **Pin** option.
![]()
Once you select **Pin**, the team will automatically move to the top of
your teams list and will be grouped into the **Pinned Teams** section.
Within the **Pinned Teams** section, you have the flexibility to reorder
the teams. Click and hold the icon on the left of each pinned team and
drag it to rearrange the order as you prefer.
![]()
## How to rename a team
Simply move your cursor over the team you want to rename. A menu icon will appear on the right side of the team's name.
Click on the menu icon. A list of options will appear. From these, select the **Rename** option.
![]()
## How to remove a team
Simply move your cursor over the team you want to remove. A menu icon will appear on the right side of the team's name.
Click on the menu icon. A list of options will appear. From these, select the **Remove** option.
![]()
## How to search for a team
To locate any team or project, regardless of how extensive your list grows, you can use the **omnibar**.
To access the omninbar you have to options, click on the **search bar at the top** or by using **keyboard shortcuts**. If you are on a Mac, press **Cmd + K**; on Windows, use **Ctrl + K**. This omnibar searches across both teams and projects.
![]()
# Export plots & charts
Source: https://docs.graphext.com/documentation/share-present-publish/export-plots-and-chart
Get publish ready charts in minutes
There are several ways to export charts in Graphext. Almost every chart
you can see throughout the application can be exported.
## Saving charts from Plot
After you've [created your plot](/documentation/data-visualization/create-a-plot), you can save it to your computer right
from the plot menu. Simply click on the "Export Chart" button at the top
and continue.
![Export chart button]()
This will present a menu from which you can select the format, adjust the size
of the final chart, toggle transparency (if supported) and including our watermark
in the final image.
![Export chart menu]()
## Saving charts from the interface
In case it is convenient, all the little charts you can see to the sides in Data
and Plot can also be exported!
![Export chart menu]()
This will bring you to the same menu previously mentioned.
## Export the Graph
Additionally, you can also export the Graph view just like any
other visualization in Graphext. Go to the Graph tab after creating
a cluster of your data and hit the "Export graph" button.
This will bring you to the same menu previously mentioned.
![Export chart menu]()
## Saving charts from Compare
In the same fashion as everywhere else, you can export any chart you find in the
Compare tab.
This will bring you to the same menu previously mentioned, albeit a bit more limited.
![Export chart menu]()
## Saving charts from Correlations
In the same fashion as everywhere else, you can export any chart you find in the
Correlations tab.
This will bring you to the same menu previously mentioned, albeit a bit more limited.
![Export chart menu]()
# Saving insights in a project
Source: https://docs.graphext.com/documentation/share-present-publish/saving-insights
Take snapshots of your data for later inspection
An [Insight](/concepts/graphext-concepts/insights) represents a snapshot of your data in the process of searching for relationships or information.
When performing a series of cross filters or a plot composition, that is a potential insight. In case you want to reproduce the state of the app at that stage, you can save an insight like this
## Saving Variables
To save an insight on how a variable reacts to another,
you can go to the options button next to the variable you are interested in,
and click save insight.
## Saving Plots
If you want to save a plot you’ve composed, you can click the “Save Insight” button.
## Saving Graphs
To save a Graph as an Insight, you can click the “Save Insight” button in the upper right corner of the Graph view.
## Saving Correlations
There are multiple ways to save correlations as insights.
You may save a group of related correlations as an insight, or individual ones.
### Saving individual correlations
In order to save a correlation chart, navigate to the Correlations tab and click on the options, just as you would in the Data tab. Then, click “Save Insight”.
### Saving a group of correlations
In order to save a group of correlations, you can click on this button on the top right to save the first 5 charts visible at that point.
This view can be customized. You can display the correlations as bubble charts, heatmaps or in boxplot mode, by clicking on the dropdown next to the save multiple insights button.
![]()
![]()
In order to change which variables appear, you can select the “Show All” dropdown, that will allow you to select any tags (link to tags) available.
Alternatively, you may select the “None” option, which will allow you to select any variables you want. Then you may save them individually or in groups of 5.
![None Selection for custom
correlation]()
For this menu to appear, you need to have
[Tags](/concepts/graphext-concepts/tags) in your variables, since it's the
only way you can filter variables in this manner at the moment.
## Saving in Compare
To save an Insight from the Compare tab, you may proceed in the same way as with the Correlations tab.
## Accessing your insights
After creating your first insight, you’ll be able to access them from the Insights tab in the navigation bar up top.
![Insights tab]()
All your insights will be presented here in chronological order of creation. You can press the play button in the lower
left corner to reproduce the chart. Any and all filters, color schemes and other settings present at the creation of the insight
will be reproduced.
On the lower right corner, you have options to edit and share the insight.
When editing an insight, you can add a title if there wasn't one, edit it, add descriptions, as well as adding or removing
the state of any cross filters that may be relevant.
You can also move all the little components and resize them. Also, when editing, in the lower left corner you may change the
insight's background theme to match whatever theme you have going on in your chart.
# Sharing an interactive project
Source: https://docs.graphext.com/documentation/share-present-publish/share-an-interactive-projects
Package and share your analysis in an interactive environment
All Graphext proyects can be made into **publications**, read-only views of the
project that have their standalone urls and can be shared safely to anyone.
This allows for very easy replication of results or further investigation on
the same data.
As an example, this is a publication of one of our projects,
the [FIFA Player Stats for the 2022 Season](https://dev-embeds.graphext.com/bc4c7002692219be/index.html). You can
view and interact with all the data, but edits are not allowed.
## Creating a new publication
To create a new publication of your project, click the options **button in the
upper left corner** and select "New Publication".
![New publication]()
This menu will pop up, which allows you to customize the environment you are
about to create.
![New publication]()
You can change:
* the title the project will display
* the author name
* what sections (tabs) you want the environment to display, which can be any of the following:
* data
* plot
* graph
* compare
* correlations
* models
* insights
* publication date and a Graphext watermark
* the theme of the publication
* whether the layout will be the same as the original project, or a simplified one with no left sidebar
* lastly, a custom Javascript input that will execute upon loading. You may need to scroll to see this.
## Modifying an existing publication
After creating a publication, you can edit it by going to the "Manage Publications" button, below the
"New Publication" one.
![Manage publications]()
You'll be presented with this menu, in which you can view all the publications made for this project,
edit them, delete them, or create a new publication.
![Manage publications menu]()
When editing or removing a publication, changes may take a few minutes to
propagate.
# Data Sources
Source: https://docs.graphext.com/faqs/data-sources
The FAQs about the supported integrations with data warehouses, APIs connections or supported type of files.
### FAQs about data sources
Yes, you can connect Graphext with your favorite data source.
We support over 15 types of integrations to ensure that we cover the needs of most people.
Some examples include Snowflake, Azure, and BigQuery.
You can read about all of them [here](/integrations/data-sources/overview).
Graphext is a versatile tool that can handle both structured and unstructured data. It allows users to perform analytics on various data types, combining unstructured data such as text or images with structured data types such as numerical, categorical, or date data.
Yes, you can write back the outputs from Graphext to the same data integrations that you can read from or to another one.
However, you need to ensure that you have writing rights.
This will enable you to automate a flow of ingesting data, applying Graphext flows (e.g., predictive models), and finally writing back the output to the same or another data warehouse.
[More information](/integrations/data-sources/overview)
Yes, you can update your projects with new data at a frequency of up to one hour. For example, you can schedule your project to fetch new data and run every morning at 7 a.m. to predict the conversion probability of the new leads that have arrived that day.
# Features
Source: https://docs.graphext.com/faqs/features
The FAQs about Graphext's features, types of analysis and limitations
### FAQs about Graphext features
1. **Adavanced Analytics**: You can analyze large datasets, gain insights and discover hidden patterns and trends
2. **Data Visualization**: You can plot your data in interactive graphs and charts. This can help you present your data in a more accessible way to your team or clients.
3. **Predictive Analysis:** You can build machine learning models to predict future trends based on past data. Some very powerful use cases are Lead Scoring or Churn Prediction Models
4. **Text Analysis**: Analyze and extract relevant information from free text, identify the main topics and keywords in just a few clicks. We use natural language processing (NLP) algorithms that you can easily apply to your reviews or any free text that you want to analyze.
5. **Social Network Analysis**: You can analyze social network structures, discover clusters, influencers, etc.
6. **Customer Segmentation**: Based on the various data of customers, you can classify them according to various categories to better target your marketing.
7. **Network Analysis**: It is used to understand and visualize complex relationships between different data points.
8. **Collaborative Analysis**: You can work with your team by sharing cases and results directly in the Graphext project. Generate reports or export inisghts in your favourite format
Yes, we offer various possibilities for sharing your insights:
1. You can **save interesting insights** within the Graphext project and share the entire project with your team so that they can explore the insights interactively and dynamically.
2. You can **export plots** or any visualization you like in all the most common formats.
3. You can export all insights as a **PDF report**.
4. You can **write back the output** of your models or clustering algorithms to your favorite **data source**.
Graphext is not a Black-box! We are committed to full transparency in Graphext. Therefore, under each transformation or flow, you can always read about every model or function that is running. If you have no data science knowledge, you can trust our good practices and expertise. However, if you are curious to know more about the algorithms or even tweak the parameters, you can do so by modifying the code of our recipe. You can read much mora about our low-code in our [API Docs](/api-docs)
All the *export\_to...* steps have a parameter *if\_exists* that allows you to choose different options to handle it. The default value is **'Fail'** to prevent you from accidentally losing your data or compromising a table's structure in your database. - **'Replace'** if you want to override the existing table. Keep in mind this option deletes your previous data. - **'Append'** if you want to append the dataset's rows to the table.
Must be one of: `"fail", "replace", "append"`
# Intro to Graphext
Source: https://docs.graphext.com/faqs/intro-to-graphext
The FAQs about Graphext and your first steps into the platform
### FAQs intro to Graphext
Graphext is an advanced data analytics and machine learning platform that
helps businesses gain valuable insights from their data through powerful analytics
(uncover patterns and trends), ML models, visualizations, and reporting capabilities.
Unlike traditional business intelligence tools, Graphext integrates advanced data analytics, machine learning and complex network analysis capabilities. This allows users to not only visualize data, but also apply sophisticated analysis and predictive modeling to derive deeper insights.
You don't have to have a technical background to use Graphext. Our product is designed to be used by everyone. We provide step by step guidance to help users create and explore their data, but we also offer an advanced “editor” so more experienced users can get more out of the product.
You can create a free account in less than minute and it doesn’t required any installation. Just sign in, and upload one your dataset or explore our examples. That is all you need to start exploring the various Graphext features.
# Performance
Source: https://docs.graphext.com/faqs/performance
The FAQs about Graphext's engine, capacity and limits.
### FAQs about Graphext performance
Graphext allows you to work seamlessly with very large datasets (many rows and columns) with minimal performance reduction, thanks to the use of advanced technology. Among many other technologies, Graphext utilizes WebAssembly, enabling real-time frontend updates. So far, we have successfully worked with projects **exceeding 30 million rows**, and we have yet to encounter a limit.
We encourage you to help us find it 😉
* Memory optimizations: All datasets are now **compressed in memory during runtime**. We've developed a custom compression algorithm based on the bitpacking technique that allows for **random access with single-value granularity**, without the need to uncompress the entire dataset or even blocks of it, while keeping overhead to a minimum.
We're now using **more memory-efficient helper data structures** for things like caching filters.
* Performance optimizations:We now take advantage of the multiple cores available in modern CPUs to speed up processing and let the browser main thread do its job, updating the UI without computation blocking it.
# Pricing and Plans
Source: https://docs.graphext.com/faqs/pricing-and-plans
The FAQs about Graphext pricing, free plan, enterprise options, etc.
### FAQs about Graphext pricing & plans
Yes, we offer a fully functional freemium version with very generous features. You can read more about our pricing [here](https://www.graphext.com/pricing-and-plans).
In addition to our generous free version, we offer a pro plan with even cooler features and almost no limitations. We also have an enterprise plan that our sales team would be extremely happy to discuss with you and adapt to your specific needs. As part of the enterprise plan we also offer professionl services for specific use cases. [Read more](https://www.graphext.com/pricing-and-plans).
At Graphext, AI features are integral to our platform, and we consider AI to be fundamental to Graphext. We've seamlessly integrated AI capabilities into our user interface, and therefore the AI features are included for **free** in the pricing.
# Security & Privacy
Source: https://docs.graphext.com/faqs/security-and-privacy
The FAQs about Graphext's privacy policies, GDPR, single-tenant options, etc
### FAQs about Graphext security and privacy policies
As a data analysis software, the user needs to transfer the data they want to analyze.
These data is stored [encrypted at rest in Google Cloud Storag](https://cloud.google.com/storage/docs/encryption).
The user has absolute control over these copies and can remove them at any time.
Our platform is hosted on Google CPD in Europe, [Belgium](https://www.google.com/about/datacenters/locations/st-ghislain/).
You can read more about the security policies [here](https://www.google.com/about/datacenters/data-security/).
Our cluster is essentially disconnected from internet with the exception of port 443 used to serve the webpage.
All web traffic is served through secure SSL connections (non-secure connections on port 80 are always redirected to https on port 443).
**Private key authentication** is required for managing our cluster.
Background processes executed by our users to do their analysis are always executed on isolated machines on a different network, so that there is no direct access to our internal systems. Nevertheless, these DBs are protected with passwords.
Datasets are stored in a private Google Cloud Storage bucket. When serving these files to an authenticated user, a signed url only valid for a very brief period of time is used.
All data is encrypted at rest.
We use **Google Cloud Audit Logs** to monitor our infrastructure.
We have successfully went through an [audit process to be compliant with GDPR](https://drive.google.com/file/d/1g2vlsWXPjSkNNDRgsEAgpuMSQZtb4sgr/view) , you can read more about our privacy policy [here](https://www.graphext.com/legal/privacy-policy).
Only after explicit consent from the user. The access to customer data through regulated interfaces is only granted to a select group of our trained employees. The primary reasons for this are to ensure effective customer support, identify and tackle security threats, troubleshoot prospective issues, and enhance data security.
The access is allocated based on the employee's role and every request for access is recorded. Only a handful of specific employees are granted access to the infrastructure. All our employees undergo privacy and security training at the start of their employment and regularly thereafter as a mandatory condition of their continued employment.
# Support & Customer Service
Source: https://docs.graphext.com/faqs/support
The FAQs about Graphext's customer support, professional services and trainnings
### FAQs about Graphext support and customer service
Yes, we include onboarding hours with the purchase of a license, and we also offer the option to increase the number of training sessions if necessary. Our team of experts is dedicated to assisting you, ensuring that Graphext delivers maximum value to your company.
Yes. We have a dedicated team of Data Scientists who provide advisory and consulting services to customers on project base. [Contact us](mailto:support@graphext.com)
# Scrape data with APIFY and analyze it with Graphext
Source: https://docs.graphext.com/tutorials/data-extraction/apify-and-graphext
Start exploring your data and discovering insights in under 5 minutes
# How To's and guides
Source: https://docs.graphext.com/tutorials/how-tos-guides/overview
A set of tutorials to get you started using Graphext
## General Overview
Overall, in depth, tutorial for starters.
## Using describe\_clusters
Using the `describe_clusters` step to sum up the overall content in a given group of topics
## The recipe for newcomers
A glimpse into a more advanced setup with Graphext
# Lead Scoring and Churn
Source: https://docs.graphext.com/tutorials/lead-scoring-churn/overview
Learn the behaviour of your clients and maximize your revenue
## Customer Churn Analysis
# Machine Learning Models
Source: https://docs.graphext.com/tutorials/machine-learning/overview
Learn how to feed data to a model and interpret its results
asdf
# Text Analysis
Source: https://docs.graphext.com/tutorials/text-analysis/overview
Discover the power of free text analysis through industry standard methods
## Analyzing Hotel Review Comments
# Tutorial Gallery
Source: https://docs.graphext.com/tutorials/tutorials
End to end guides to help you get the maximum value from Graphext
Here we compile a collection of tutorials made by the team to help you get the most out of the tool.
## General Overview
Overall, in depth, tutorial for starters.
## Text analysis through review comments
Analyzing hotel review comments and extracting information from free text data.
## Using describe\_clusters
Using the `describe_clusters` step to sum up the overall content in a given group of topics
## The recipe for newcomers
A glimpse into a more advanced setup with Graphext