🔄 Distribution-Aware Encoding

Distribution-Aware Encoding

Automatically detect and handle various data distributions for optimal preprocessing.

📋 Overview

The Distribution-Aware Encoder is a powerful preprocessing layer that automatically detects and handles various data distributions. It intelligently transforms your data while preserving its statistical properties, leading to better model performance.

🔍

Automatic Distribution Detection

Identifies data patterns using statistical analysis

⚙️

Smart Transformations

Applies distribution-specific preprocessing

🚀

Production-Ready

Built with pure TensorFlow operations for deployment

🔌

Flexible Integration

Works seamlessly with KDP's preprocessing pipeline

📊

Graph Mode Compatible

Works in both eager and graph execution modes

📦

Memory Efficient

Optimized for large-scale datasets

🎯 Use Cases

💰

Financial Data

Handling heavy-tailed distributions in price movements

📡

Sensor Data

Processing periodic patterns in time series

👤

User Behavior

Managing sparse data with many zeros

🌍

Natural Phenomena

Handling multimodal distributions

🔢

Count Data

Processing discrete and zero-inflated distributions

🚀 Getting Started

Basic Usage

from kdp import PreprocessingModel, FeatureType

# Define numerical features
features_specs = {
    "price": FeatureType.FLOAT_NORMALIZED,
    "volume": FeatureType.FLOAT_RESCALED,
    "rating": FeatureType.FLOAT_NORMALIZED
}

# Initialize model with distribution-aware encoding
preprocessor = PreprocessingModel(
    path_data="data/my_data.csv",
    features_specs=features_specs,
    use_distribution_aware=True,  # Enable distribution-aware encoding
    distribution_aware_bins=1000  # Number of bins for distribution analysis
)

Advanced Configuration

from kdp import FeatureType, PreprocessingModel

from kdp.features import NumericalFeature
from kdp.layers.distribution_aware_encoder_layer import DistributionType

features_specs = {
    "price": NumericalFeature(
        name="price",
        feature_type=FeatureType.FLOAT_NORMALIZED,
        preferred_distribution=DistributionType.LOG_NORMAL  # Specify distribution
    ),
    "volume": NumericalFeature(
        name="volume",
        feature_type=FeatureType.FLOAT_RESCALED,
        preferred_distribution=DistributionType.ZERO_INFLATED  # Handle sparse data
    )
}

preprocessor = PreprocessingModel(
    path_data="data/my_data.csv",
    features_specs=features_specs,
    use_distribution_aware=True,
    distribution_aware_bins=1000,
)

📊 Supported Distributions

Distribution Type	Description	Detection Criteria	Use Case
Normal	Standard bell curve	Skewness < 0.5, Kurtosis ≈ 3.0	Height, weight measurements
Heavy-Tailed	Longer tails than normal	Kurtosis > 4.0	Financial returns
Multimodal	Multiple peaks	Multiple histogram peaks	Mixed populations
Uniform	Even distribution	Bounded between 0 and 1	Random sampling
Exponential	Exponential decay	Positive values, skewness > 1.0	Time between events
Log-Normal	Normal after log transform	Positive values, skewness > 2.0	Income distribution
Discrete	Finite distinct values	Low unique value ratio (< 0.1)	Count data
Periodic	Cyclic patterns	Significant autocorrelation	Seasonal data
Sparse	Many zeros	Zero ratio > 0.5	User activity data
Beta	Bounded with shape parameters	Bounded [0,1], skewness > 0.5	Proportions
Gamma	Positive, right-skewed	Positive values, mild skewness	Waiting times
Poisson	Count data	Discrete positive values	Event counts
Cauchy	Extremely heavy-tailed	Very high kurtosis (> 10.0)	Extreme events
Zero-Inflated	Excess zeros	Moderate zero ratio (0.3-0.5)	Rare events
Bounded	Known bounds	Explicit bounds provided	Physical measurements
Ordinal	Ordered categories	Discrete ordered values	Ratings, scores

⚙️ Configuration Options

On PreprocessingModel

These two are the whole model-level surface:

Parameter	Type	Default	Description
use_distribution_aware	bool	False	Enable distribution-aware encoding
distribution_aware_bins	int	1000	Number of bins for distribution analysis

Per feature, set preferred_distribution on a NumericalFeature to skip automatic detection and force a specific distribution.

Automatic detection is a heuristic, and it is not always right

Measured over six seeded samples, detection identifies heavy_tailed, log_normal, discrete, periodic, sparse and beta correctly and consistently. Four shapes are consistently confused with a neighbour:

Actual	Detected as
normal	multimodal
uniform	multimodal
multimodal	periodic
exponential	log_normal

The encoding still works — a neighbouring distribution's transform is usually a reasonable choice — but if a column's shape matters to you, set preferred_distribution on the feature rather than relying on detection. test/layers/test_distribution_aware_encoder.py pins this behaviour, so it cannot change without being noticed.

Available transformations

transform_type accepts these; anything else raises. auto is the default on the encoder and picks from the rest using the column's own shape — whether it is bounded, contains zeros, or contains negative values.

Name	Requires	Use for
auto	—	Let KDP choose from the column's shape.
none	—	Pass the values through unchanged.
log	Strictly positive	Long right tails, such as income.
sqrt	Non-negative	Milder right skew; tolerates zeros.
cube-root	Any sign	Skew in data that also goes negative.
arcsinh	Any sign	Log-like compression that accepts zero and negatives.
box-cox	Strictly positive	Power transform toward normality.
yeo-johnson	Any sign	Box-Cox for data including zero and negatives.
logit	Values inside (0, 1)	Proportions and rates.
min-max	—	Rescale to a fixed range.
robust-scale	—	Median and IQR per feature; resists outliers.
quantile	—	Rank-based, per feature; flattens any shape.

auto narrows the candidates to what the data allows

A column with zeros never gets log, and one with negative values never gets box-cox, so auto cannot produce infinities from an unsuitable transform. Restrict the search yourself with auto_candidates=["log", "sqrt"].

On the DistributionAwareEncoder layer

The processor builds this layer for you with detect_periodicity=True, handle_sparsity=True, adaptive_binning=True and mixture_components=3 fixed. To change them, use the layer directly through a custom preprocessing pipeline — passing them to PreprocessingModel raises TypeError.

Parameter	Type	Default	Description
detect_periodicity	bool	True	Detect and handle periodic patterns
handle_sparsity	bool	True	Special handling for sparse data
adaptive_binning	bool	None	Learn bin boundaries from the batch rather than using fixed ones
embedding_dim	int	None	Output dimension for feature projection
add_distribution_embedding	bool	False	Add learned distribution type embedding
epsilon	float	1e-6	Small value to prevent numerical issues
transform_type	str	"auto"	Type of transformation to apply

🎯 Best Practices

Distribution Detection

• Start with automatic detection
• Specify preferred distributions only when confident
• Use appropriate bin sizes for your data scale
• Monitor detection accuracy with known distributions

Performance Optimization

• Enable periodic detection for time series data
• Use sparse handling for data with many zeros
• Consider memory usage with large bin sizes
• Use appropriate embedding dimensions

Integration Tips

• Combine with other KDP features for best results
• Use appropriate feature types (FLOAT_NORMALIZED, FLOAT_RESCALED)
• Monitor model performance with different configurations
• Consider using distribution embeddings for complex patterns

🔍 Examples

Financial Data Processing

from kdp import PreprocessingModel, FeatureType
from kdp.features import NumericalFeature
from kdp.layers.distribution_aware_encoder_layer import DistributionType

# Define financial features
features_specs = {
    "price": NumericalFeature(
        name="price",
        feature_type=FeatureType.FLOAT_NORMALIZED,
        preferred_distribution=DistributionType.LOG_NORMAL
    ),
    "volume": NumericalFeature(
        name="volume",
        feature_type=FeatureType.FLOAT_RESCALED,
        preferred_distribution=DistributionType.ZERO_INFLATED
    ),
    "volatility": NumericalFeature(
        name="volatility",
        feature_type=FeatureType.FLOAT_NORMALIZED,
        preferred_distribution=DistributionType.CAUCHY
    )
}

# Create preprocessing model
preprocessor = PreprocessingModel(
    path_data="data/financial_data.csv",
    features_specs=features_specs,
    use_distribution_aware=True,
    distribution_aware_bins=1000,
    embedding_dim=32,        # Project to fixed dimension
)

Sensor Data Processing

from kdp import DistributionType, FeatureType, NumericalFeature, PreprocessingModel

features_specs = {
    "temperature": NumericalFeature(
        name="temperature",
        feature_type=FeatureType.FLOAT_NORMALIZED,
        preferred_distribution=DistributionType.NORMAL
    ),
    "humidity": NumericalFeature(
        name="humidity",
        feature_type=FeatureType.FLOAT_NORMALIZED,
        preferred_distribution=DistributionType.BETA  # Bounded between 0-100%
    ),
    "pressure": NumericalFeature(
        name="pressure",
        feature_type=FeatureType.FLOAT_NORMALIZED,
        preferred_distribution=DistributionType.NORMAL
    )
}

preprocessor = PreprocessingModel(
    path_data="data/sensor_data.csv",
    features_specs=features_specs,
    use_distribution_aware=True,
    distribution_aware_bins=500,  # Fewer bins for simpler distributions
    embedding_dim=16             # Smaller embedding for simpler patterns
)

📊 Model Architecture

The distribution-aware encoder architecture automatically adapts to your data's distribution, transforming numerical features to better match their underlying statistical properties, improving model performance.

🔗 Related Topics

🔢 Numerical Features 🧮 Advanced Numerical Embeddings 👁️ Tabular Attention 🎯 Feature Selection

---

← Cross Features Advanced Numerical Embeddings →