> ## Documentation Index
> Fetch the complete documentation index at: https://docs.graphext.com/llms.txt
> Use this file to discover all available pages before exploring further.

# embed_text

> Parse and calculate a (word-averaged) embedding vector for each text. 

An embedding vector is a numerical representation of a text, such that different numerical components of the vector
capture different dimensions of the text's meaning. Embeddings can be used, for example, to calculate the *semantic similarity*
between pairs of texts (see `link_embeddings`, for example, to create a network of texts connected by similarity).

In this step, embeddings of texts are calculated as (weighted) averages of the embeddings of each text's individual
words (the individual word embeddings are [GloVe](https://nlp.stanford.edu/projects/glove/) vectors, as provided by
[spaCy's](https://spacy.io/) [language models](https://spacy.io/models/en#en_core_web_md)).

Use either the `language` *parameter* or a second input *column* to specify the language of the input texts. If neither
is provided, the language will be inferred automatically from the texts themselves (which is equivalent to first creating
a language column using the `infer_language` step).

## Usage

The following example shows how the step can be used in a recipe.

<Accordion title="Examples" icon="code" defaultOpen="true">
  <Tabs>
    <Tab title="Example 1">
      To calculate embeddings in a way that emphasizes entities (recognized products, people etc.) over regular words:

      ```stan theme={null}
      embed_text(ds.text, ds.lang, {"weighted": true}) -> (ds.embedding)
      ```
    </Tab>

    <Tab title="Signature">
      General syntax for using the step in a recipe. Shows the inputs and outputs the step is expected to receive and will produce respectively. For futher details see sections below.

      ```stan theme={null}
      embed_text(text: text, *lang: category, {
          "param": value,
          ...
      }) -> (embedding: list[number])
      ```
    </Tab>
  </Tabs>
</Accordion>

## Inputs & Outputs

The following are the inputs expected by the step and the outputs it produces. These are generally
columns (`ds.first_name`), datasets (`ds` or `ds[["first_name", "last_name"]]`) or models (referenced
by name e.g. `"churn-clf"`).

<Accordion title="Inputs" icon="right-to-bracket">
  <ParamField path="text" type="column[text]" required>
    A text column to calculate embeddings for.
  </ParamField>

  <ParamField path="*lang" type="column[category]">
    An (optional) column identifying the languages of the corresponding texts. It is used to identify the correct model (spaCy)
    to use for each text. If the dataset doesn't contain such a column yet, it can be created using the `infer_language` step.
    Ideally, languages should be expressed as two-letter
    [ISO 639-1 language codes](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes), such as "en", "es" or "de" for
    English, Spanish or German respectively. We also detect fully spelled out names such as "english", "German", "allemande"
    etc., but it is not guaranteed that we will recognize all possible spellings correctly always, so ISO codes should be
    preferred.

    Alternatively, if all texts are in the same language, it can be identified with the `language` *parameter* instead.
  </ParamField>
</Accordion>

<Accordion title="Outputs" icon="right-from-bracket">
  <ParamField path="embedding" type="column[list[number]]" required>
    A column of embedding vectors capturing the meaning of each input text.
  </ParamField>
</Accordion>

## Configuration

The following parameters can be used to configure the behaviour of the step by including them in
a json object as the last "input" to the step, i.e. `step(..., {"param": "value", ...}) -> (output)`.

<Accordion title="Parameters" defaultOpen="true" icon="sliders">
  <ParamField path="embedding" type="object">
    Configure how embeddings are calculated.
    Toggle word vector weighting and normalization.

    <Accordion title="Properties">
      <ParamField path="weighted" type="boolean" default="true">
        Whether entities have more influence on the embedding than regular words.
      </ParamField>

      <ParamField path="normalized" type="boolean" default="true">
        Whether to normalize embeddings. Each will have a length (norm) of 1.0.
      </ParamField>
    </Accordion>
  </ParamField>

  <ParamField path="extended_language_support" type="boolean" default="false">
    Whether to enable support for additional languages.
    By default, Arabic ("ar"), Catalan ("ca"), Basque ("eu"), and Turkish ("tu") are not enabled,
    since they're supported only by a different class of language models (stanfordNLP's Stanza)
    that is much slower than the rest. This parameter can be used to enable them.
  </ParamField>

  <ParamField path="min_language_freq" type="[number, integer]" default="0.02">
    Minimum number (or proportion) of texts to include a language in processing.
    Any texts in a language with fewer documents than these will be ignored. Can be useful to speed up
    processing when there is noise in the input languages, and when ignoring languages with a small number of
    documents only is acceptable. Values smaller than 1 will be interpreted as a *proportion* of all texts, and
    values greater than or equal to 1 as an *absolute number* of documents.

    <Accordion title="Options">
      <Tabs>
        <Tab title="number">
          <ParamField path="{_}" type="number">
            number.

            Values must be in the following range:

            ```javascript theme={null}
            0 < {_} < 1
            ```
          </ParamField>
        </Tab>

        <Tab title="integer">
          <ParamField path="{_}" type="integer">
            integer.

            Values must be in the following range:

            ```javascript theme={null}
            1 ≤ {_} < inf
            ```
          </ParamField>
        </Tab>
      </Tabs>
    </Accordion>
  </ParamField>

  <ParamField path="language" type="[string, null]">
    The language of inputs texts.
    If all texts are in the same language, it can be specified here instead of passing it as an input column. The language will be used to identify the correct spaCy model to parse and analyze the texts. For allowed values, see the comment regarding the `lang` column above.
  </ParamField>
</Accordion>