📝 Text Features

Text Features in KDP

Vocabulary-based text vectorization, learned from your own data.

📋 Overview

KDP builds a vocabulary from the text column during the statistics pass, then encodes each row with Keras TextVectorization against that vocabulary. Everything is learned from your corpus — there are no downloaded embeddings and no external model weights.

📝 Basic Usage

from kdp import PreprocessingModel, FeatureType

preprocessor = PreprocessingModel(
    path_data="reviews.csv",
    features_specs={
        "review_text": FeatureType.TEXT,
    },
)
preprocessor.build_preprocessor()

By default a text column becomes a 35-token integer sequence, padded with zeros.

⚙️ Configuration Parameters

TextFeature forwards its keyword arguments to Keras TextVectorization, apart from stop_words, which KDP applies itself beforehand.

Parameter	Type	Default	Description
`stop_words`	list[str]	`[]`	Words stripped before vectorization, by KDP's own text preprocessing layer.
`output_sequence_length`	int	`35`	Token count per row, and therefore the output width. Applies to `output_mode="int"` only.
`output_mode`	str	`"int"`	`"int"`, `"multi_hot"` or `"count"`. See the table below.
`max_tokens`	int	—	Caps the vocabulary, counting the out-of-vocabulary slot (and the padding slot under `"int"`). When the cap is smaller than the vocabulary the statistics found, the vectorizer is adapted on the column so the words kept are the most frequent ones.
`ngrams`	int \| tuple	`None`	Generate n-grams in addition to single tokens. The statistics collect single words only, so setting this reads the column again to build a vocabulary that holds the n-grams.
`split`, `standardize`	str \| callable	Keras defaults	Passed straight through to `TextVectorization`. Either one changes what a token is, so setting them also reads the column again rather than reusing the collected vocabulary.

Pretrained embeddings and attention are not implemented

Earlier documentation advertised use_pretrained, pretrained_name (GloVe, word2vec, BERT), tokenizer, use_attention, attention_heads, attention_dropout, max_sequence_length, embedding_dim and sequence_length. None of these exist. TextFeature accepts any keyword without complaint, so they appear to work while changing nothing — verified by comparing model output with and without each one. KDP learns its vocabulary from your data; it does not download or load pretrained language models. To use one, wrap it yourself with custom preprocessing.

🔤 Output Modes

Mode	Output width	What each value means
`"int"` (default)	`output_sequence_length`	Token index at that position; order is preserved.
`"multi_hot"`	vocabulary size	1 if the token appears anywhere in the row, else 0. Order is discarded.
`"count"`	vocabulary size	How many times the token appears in the row.

from kdp import PreprocessingModel
from kdp.features import FeatureType, TextFeature

# Bag-of-words instead of a padded sequence
preprocessor = PreprocessingModel(
    path_data="reviews.csv",
    features_specs={
        "review_text": TextFeature(
            name="review_text",
            feature_type=FeatureType.TEXT,
            output_mode="multi_hot",
        ),
    },
)
preprocessor.build_preprocessor()

tf_idf needs weights KDP does not compute

output_mode="tf_idf" requires an IDF weight array alongside the vocabulary. KDP's statistics pass records the vocabulary only, so this mode raises a clear Keras error rather than working. Use "count" and apply your own weighting downstream if you need it.

🧹 Stop Words

TextFeature(
    name="review_text",
    feature_type=FeatureType.TEXT,
    stop_words=["the", "a", "an", "and", "or"],
    output_sequence_length=64,
)

Stop words are removed before vectorization, so they never enter the vocabulary and never occupy a token slot.

🔗 Combining With Other Features

N-grams for short text

TextFeature(
    name="product_title",
    feature_type=FeatureType.TEXT,
    ngrams=2,                       # unigrams and bigrams
    output_sequence_length=24,
)

Feature selection over text

from kdp import FeatureType, PreprocessingModel

preprocessor = PreprocessingModel(
    path_data="reviews.csv",
    features_specs={
        "review_text": FeatureType.TEXT,
        "rating": FeatureType.FLOAT_NORMALIZED,
    },
    feature_selection_placement="text",   # or "all_features"
)

💎 Practical Notes

Sequence length drives width

In "int" mode the output is exactly output_sequence_length columns wide. Long default sequences on short text are mostly padding.

Text needs a statistics pass

The vocabulary comes from your data, so path_data is required and the column is read end to end.

multi_hot for keyword signals

When only presence matters — tags, short titles — "multi_hot" is smaller and easier to learn from than a padded sequence.

Tokens are standardized before they are counted

Text is lowercased and stripped of punctuation before it is split on whitespace, so "Great product," contributes great and product. That is TextVectorization's own default, and the vocabulary collected from your data is spelled to match it.