DataSet

The DataSet class (exposed as modeva.DataSet) handles data loading, exploration, preprocessing, feature selection, outlier detection, splitting, drift and sampling.

class modeva.data.local_dataset.LocalDataSet(name: str = 'dataset', experiment_name: str = 'default')

bin_numerical(features: str | Tuple = None, dataset: str = 'main', method: str = 'uniform', bins: int | Dict = 10)

Performs binning transformation on numerical features by discretizing continuous values into discrete bins.

This function transforms continuous numerical features into discrete bins using various binning strategies. The binning configuration is stored internally and can be used for both forward and inverse transformations. Note that this preprocessing step can only be called once; subsequent calls will overwrite previous configurations.

Parameters:

featuresstr or tuple, default=None

Names of features to be binned. If None, all numerical features in the dataset will be processed.

dataset{“main”, “train”, “test”}, default=”main”

Specifies which dataset partition to use for generating the binning boundaries.

method{“uniform”, “quantile”, “precompute”}, default=”uniform”

Binning strategy to use:

• “uniform”: Creates bins of equal width

• “quantile”: Creates bins with equal number of samples

• “precompute”: Uses manually specified bin boundaries

binsint or dict, default=10

Controls binning granularity:

• If int: Number of bins for numerical features. For “quantile”, this is the maximum number of bins. For “auto-xgb1”, this sets XGBoost’s max_bin parameter.

• If dict: Manual bin specifications for each feature, only used with method=”precompute”. Format: {feature_name: array_of_bin_edges}. Example: {“X0”: [0.1, 0.5, 0.9]} Note: Cannot specify bins for categorical features.

Returns:

ValidationResult

A container object with the following components:

• key: “data_preprocess_binning”

• data: Name of the dataset used

• inputs: Dictionary of input parameters

• value: Dictionary containing binning configuration for each feature:

  • “fidx”: Feature index

  • “bins”: Bin boundaries

  • “feature_names_out”: Output feature names

Examples

data_drift_test(dataset1: str = 'train', dataset2: str = 'test', sample_idx1: Tuple[int] | ndarray = None, sample_idx2: Tuple[int] | ndarray = None, name1: str = None, name2: str = None, distance_metric: str = 'PSI', psi_method: str = 'uniform', psi_bins: int = 10)

Evaluates the distributional differences between two data samples using various distance metrics.

This function compares two datasets by calculating the distance metrics between them, allowing for the assessment of data drift. Only features in X and y are compared.

Parameters:

dataset1{“main”, “train”, “test”}, default=”train”

Identifier for the first dataset to analyze. Must be one of the predefined dataset types.

dataset2{“main”, “train”, “test”}, default=”test”

Identifier for the second dataset to analyze. Must be one of the predefined dataset types.

sample_idx1Tuple or np.ndarray of int, default=None

Indices of samples to select from the first dataset. If None, will use all samples in the specified data.

sample_idx2Tuple or np.ndarray of int, default=None

Indices of samples to select from the second dataset. If None, will use all samples in the specified data.

name1str, default=None

Custom label for the first dataset used in visualization outputs. If None, will set to dataset1.

name2str, default=None

Custom label for the second dataset used in visualization outputs. If None, will set to dataset2.

distance_metric{“PSI”, “WD1”, “KS”}, default=”PSI”

Method to calculate distribution difference:

• “PSI”: Population Stability Index

• “WD1”: Wasserstein Distance

• “KS”: Kolmogorov-Smirnov test

psi_method{“uniform”, “quantile”}, default=”uniform”

Binning strategy for PSI calculation. Only applicable when distance_metric=”PSI”.

psi_binsint, default=10

Number of bins for PSI calculation. Only applicable when distance_metric=”PSI”.

Returns:

ValidationResult

A container object with the following components:

• key: “data_drift”

• data: Name of the dataset used

• inputs: Dictionary of input parameters used in the analysis

• value: Dictionary containing:

  • “Distance_Scores”: Feature-wise distance metrics between the two datasets

• table: DataFrame representation of the distance scores

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

  • “summary”: Horizontal bar plot visualizing the distance metric for each feature.

  • “(“density”, <feature_name>)”: Density distribution comparison plots for two set of samples.

Examples

delete_extra_data(name: str)

Delete a group of data.

Protected features will also be deleted if specified.

Parameters:

namestr

The name of this data group, e.g., “oot” (stands for out of time data).

delete_registered_data(name: str)

Delete a registered dataset from MLflow.

Parameters:

namestr

Name of the registered dataset to delete.

Raises:

MlflowException

If the dataset doesn’t exist or there are permission issues.

detect_outlier_cblof(dataset: str = 'main', threshold: float = 0.99, method: str = 'kmeans', n_clusters: int = 10, cluster_threshold: float = 0.1, use_weights: bool = False, random_state: int = 0)

Performs Cluster-Based Local Outlier Factor (CBLOF) detection on the dataset.

CBLOF is an outlier detection method that uses clustering to identify anomalies. It first partitions data into small and large clusters, then calculates outlier scores based on the distances between points and cluster centers, optionally weighted by cluster sizes.

Parameters:

dataset{“main”, “train”, “test”}, default=”main”

Specifies which dataset partition to analyze for outlier detection.

thresholdfloat, default=0.99

Quantile threshold for determining outliers. Values between 0 and 1, where higher values result in fewer outliers.

method{“kmeans”, “gmm”}, default=”kmeans”

Clustering algorithm to use: K-means or Gaussian Mixture Model.

n_clustersint, default=10

Number of clusters to partition the data into.

cluster_thresholdfloat, default=0.1

Proportion threshold to distinguish between small and large clusters. Clusters with proportion of points above this are considered large.

use_weightsbool, default=False

Whether to weight outlier scores by cluster sizes.

random_stateint, default=0

Random seed for reproducibility in clustering algorithms.

Returns:

ValidationResult

A container object with the following components:

• key: “data_outlier_cblof”

• data: Name of the dataset used

• inputs: Dictionary of input parameters

• table: Dictionary containing:

  • “outliers”: DataFrame of detected outlier samples

  • “non-outliers”: DataFrame of normal samples

• func: Callable function that computes outlier scores for new data

• options: Dictionary of visualizations configuration for a histogram plot where x-axis is the outlier scores, and y-axis is the density. Run results.plot() to show this plot.

Examples

detect_outlier_isolation_forest(dataset: str = 'main', threshold: float = 0.99, n_estimators: int = 100)

Performs outlier detection using the Isolation Forest algorithm.

This method implements outlier detection by training an Isolation Forest model on the specified dataset. It preprocesses the data, fits the model, calculates outlier scores, and returns both the detection results and visualization options. The method uses score thresholding based on quantiles to identify outliers.

Parameters:

dataset{“main”, “train”, “test”}, default=”main”

Specifies which dataset partition to analyze for outliers.

thresholdfloat, default=0.99

Quantile threshold for outlier classification. Values between 0 and 1, where higher values result in fewer outliers being identified.

n_estimatorsint, default=100

Number of trees in the Isolation Forest ensemble. Higher values generally provide better stability but increase computation time.

Returns:

ValidationResult

A container object with the following components:

• key: “data_outlier_isolationforest”

• data: Name of the dataset used

• inputs: Dictionary of input parameters

• table: Dictionary containing:

  • outliers: DataFrame of identified outlier samples

  • non-outliers: DataFrame of normal samples

• func: Callable function that computes outlier scores for new data

• options: Dictionary of visualizations configuration for a histogram plot where x-axis is the outlier scores, and y-axis is the density. Run results.plot() to show this plot.

Examples

detect_outlier_pca(dataset: str = 'main', threshold: float = 0.99, cumulative_variance_threshold: float = 0.9, method: str = 'mahalanobis', sparse_pca: bool = False, alpha: float = 1.0, random_state: int = 0)

Performs outlier detection using PCA-based methods.

This function implements two PCA-based outlier detection approaches: Mahalanobis distance in PCA space and reconstruction error using PCA components. It first transforms the data using PCA (or Sparse PCA), then calculates outlier scores based on the chosen method, and identifies outliers using a threshold on these scores.

Parameters:

dataset{“main”, “train”, “test”}, default=”main”

Dataset to analyze for outliers. Options are “main” (full dataset), “train” (training set), or “test” (test set).

thresholdfloat, default=0.99

Quantile threshold for outlier classification. Values above this quantile of outlier scores are considered outliers. Must be between 0 and 1.

cumulative_variance_thresholdfloat, default=0.9

Threshold for selecting number of principal components based on cumulative explained variance. Must be between 0 and 1.

method{“mahalanobis”, “reconst_error”}, default=”mahalanobis”

Method for calculating outlier scores:

• “mahalanobis”: Uses Mahalanobis distance in PCA space

• “reconst_error”: Uses reconstruction error with PCA components

sparse_pcabool, default=False

Whether to use Sparse PCA instead of standard PCA for dimensionality reduction.

alphafloat, default=1.0

Sparsity controlling parameter for Sparse PCA. Only used when sparse_pca=True.

random_stateint, default=0

Random seed for reproducibility.

Returns:

ValidationResult

A container object with the following components:

• key: “data_outlier_pca”

• data: Name of the dataset used

• inputs: Dictionary of input parameters

• table: Dictionary containing:

  • “outliers”: DataFrame of detected outlier samples

  • “non-outliers”: DataFrame of normal samples

• func: Callable function that computes outlier scores for new data

• options: Dictionary of visualizations configuration for a histogram plot where x-axis is the outlier scores, and y-axis is the density. Run results.plot() to show this plot.

Examples

eda_1d(feature: str, dataset: str = 'main', plot_type: str = 'density', bins: int = 10, sample_size: int = None, random_state: int = 0)

Creates a univariate visualization for analyzing the distribution of a single feature.

This function generates either a density plot, histogram, or bar chart depending on the feature type (numerical or categorical) and user preferences. It supports data sampling for large datasets and provides customizable visualization options including bin sizes for histograms.

Parameters:

featurestr

Name of the feature to visualize in the dataset.

dataset{“main”, “train”, “test”}, default=”main”

Specifies which dataset partition to use for visualization.

plot_type{“density”, “histogram”}, default=”density”

Type of visualization for numerical features. Categorical features always use bar charts.

binsint, default=10

Number of bins for histogram visualization when plot_type is “histogram”.

sample_sizeint, default=None

Size of random sample to use for visualization. Uses all data if None.

random_stateint, default=0

Random seed for reproducible sampling when sample_size is specified.

Returns:

ValidationResult

A container object with the following components:

• key: “data_eda_1d”

• data: Name of the dataset used

• inputs: Dictionary of input parameters used

• options: Dictionary of visualizations configuration for a bar / histogram / density plot. Run results.plot() to show this plot.

Examples

eda_2d(feature_x: str, feature_y: str, feature_color: str = None, dataset: str = 'main', sample_size: int = None, smoother_order: int = None, random_state: int = 0)

Creates a bivariate visualization between two features with optional color encoding.

Generates various types of plots (scatter, box, or bar) based on the feature types (numerical or categorical) to visualize the relationship between two features. The visualization can be enhanced with a third feature through color coding, and includes options for data sampling and smoothing.

Parameters:

feature_xstr

Name of the feature to be plotted on the x-axis.

feature_ystr

Name of the feature to be plotted on the y-axis.

feature_colorstr, default=None

Name of the feature used for color encoding in scatter plots. If None, no color encoding is applied.

dataset{“main”, “train”, “test”}, default=”main”

Specifies which dataset partition to use for visualization.

sample_method{“random”}, default=’random’

Method used for data sampling. Currently only supports random sampling.

sample_sizeint, default=None

Maximum number of samples to use in visualization. If None, uses all available data.

smoother_orderint, default=None

Order of polynomial for trend line smoothing. If None, no smoothing curve is drawn.

random_stateint, default=0

Random seed for reproducible sampling.

Returns:

ValidationResult

A container object with the following components:

• key: “data_eda_2d”

• data: Name of the dataset used

• inputs: Dictionary of input parameters

• options: Dictionary of visualizations configuration for a scatter / box / stacked bar plot. Run results.plot() to show this plot.

Examples

eda_3d(feature_x: str, feature_y: str, feature_z: str, feature_color: str = None, dataset: str = 'main', sample_size: int = 1000, random_state: int = 0)

Creates an interactive 3D scatter plot visualization for exploring relationships between three features.

This function generates a 3D scatter plot using the specified features, with an optional fourth feature represented by color. It supports subsampling for large datasets and handles both numerical and categorical features for coloring. The visualization is powered by mocharts library and provides interactive features like tooltips and adjustable viewports.

Parameters:

feature_xstr

Name of the feature to be plotted on the x-axis.

feature_ystr

Name of the feature to be plotted on the y-axis.

feature_zstr

Name of the feature to be plotted on the z-axis.

feature_colorstr, optional

Name of the feature used for coloring points. If numerical, creates a color gradient; if categorical, creates distinct colors per category.

dataset{“main”, “train”, “test”}, default=”main”

Specifies which dataset partition to visualize.

sample_method{“random”}, default=”random”

Method used for subsampling the data. Currently only supports random sampling.

sample_sizeint, optional, default=1000

Maximum number of points to plot. If dataset is larger, points will be randomly sampled. Set to None to use all points.

random_stateint, default=0

Seed for random number generator used in sampling.

Returns:

ValidationResult

A container object with the following components:

• key: “data_eda_3d”

• data: Name of the dataset used

• inputs: Dictionary of input parameters

• options: Dictionary of visualizations configuration for a 2D scatter plot. Run results.plot() to show this plot.

Examples

eda_correlation(features: Tuple | List = None, dataset: str = 'main', method: str = 'pearson', sample_size: int = None, random_state: int = 0)

Calculate and visualize correlation matrices between features in the dataset.

Pearson / Spearman / Kendall / XI correlation for two numerical features. Symmetric Theil’s U correlation for two categorical features. Correlation ratio for numerical and categorical features.

Parameters:

featurestuple, default=None

Feature names to include in correlation analysis. If None, all features in the dataset will be used.

dataset{“main”, “train”, “test”}, default=”main”

Specifies which dataset partition to use for correlation analysis.

method{“pearson”, “spearman”, “kendall”, “xicor”}, default=”pearson”

The algorithm for calculating the correlation between two numerical features.

• “pearson”: Pearson correlation measures the linear relationship between two continuous variables. Its value ranges from −1 (perfect negative linear relationship) to 1 (perfect positive linear relationship), with 0 indicating no linear correlation. It is sensitive to linear relationships but not to nonlinear patterns.

• “spearman”: Spearman correlation assesses the strength and direction of a monotonic relationship between two variables, based on their ranks. It ranges from −1 to 1, where 1 indicates a perfect increasing monotonic relationship and −1 a perfect decreasing one. It is robust to outliers and can capture non-linear relationships.

• “kendall”: Kendall Tau measures the association between two ranked variables, focusing on the consistency of the order between them. Its value ranges from −1 (perfect discordance) to 1 (perfect concordance). It is particularly useful for ordinal data and is robust to outliers.

• “xicor”: XiCor detects both linear and nonlinear dependencies between continuous variables. It typically ranges from 0 (no dependence) to 1 (strong dependence), providing a more comprehensive view of relationships. Negative XI correlation does not have any innate significance, other than close to zero. See details in the paper [1].

[1] Chatterjee, S. (2021). A new coefficient of correlation. J. Amer. Statist. Assoc., 116, no. 536, 2009–2022.

sample_sizeint, default=None

Number of random samples to use for calculation. If None, uses entire dataset. Useful for large datasets to reduce computation time.

random_stateint, default=0

Random seed for reproducible sampling when sample_size is specified.

Returns:

ValidationResult

A container object with the following components:

• key: “data_eda_correlation”

• data: Name of the dataset used

• inputs: Dictionary of input parameters

• table: Correlation matrix as a pandas DataFrame

• options: Dictionary of visualizations configuration for a heatmap plot. Run results.plot() to show this plot.

Examples

eda_pca(features: Tuple | List = None, n_components: int = None, dataset: str = 'main', sample_size: int = None, categorical_encoding: str = 'ordinal', random_state: int = 0)

Performs Principal Component Analysis (PCA) on specified features with preprocessing for both numerical and categorical variables.

This function handles the complete PCA workflow, including data preprocessing (standardization for numerical features and encoding for categorical features), PCA computation, and visualization of results through loadings and explained variance.

Parameters:

featurestuple, default=None

Features to include in PCA analysis. If None, all available features are used.

n_componentsint, default=None

Number of principal components to compute. If None, equals the number of features.

dataset{“main”, “train”, “test”}, default=”main”

Specifies which dataset partition to use for the analysis.

sample_sizeint, default=None

Size of random subsample to use for computation. If None or if data size is smaller, uses full dataset.

categorical_encoding{“ordinal”, “onehot”}, default=”ordinal”

Method for encoding categorical variables:

• “ordinal”: Converts categories to integer values

• “onehot”: Creates binary columns for each category (minus one reference category)

random_stateint, default=0

Seed for random operations, ensuring reproducibility.

Returns:

ValidationResult

A container object with the following components:

• key: “data_eda_pca”

• data: Name of the dataset used

• inputs: Dictionary of input parameters

• table: DataFrame containing PCA transformed data

• options: Dictionary of visualizations configuration for a PCA loadings and explained variance plot. Run results.plot() to show this plot.

Examples

eda_umap(features: Tuple | List = None, n_neighbors: int = 15, n_components: int = 2, metric: str = 'euclidean', dataset: str = 'main', sample_size: int = None, categorical_encoding: str = 'ordinal', random_state: int = 0)

Performs UMAP dimensionality reduction on the specified features of the dataset.

This function processes both numerical and categorical features, applies appropriate encoding and scaling, and then performs UMAP dimensionality reduction. It handles data preprocessing including optional subsampling, categorical encoding, and numerical scaling before applying the UMAP algorithm.

Parameters:

featurestuple, default=None

Names of features to use for UMAP. If None, all features are used.

n_neighborsint, default=15

Number of neighboring points used in local approximations. Values typically range from 2 to 100, with larger values producing more global structure.

n_componentsint, default=3

Number of dimensions in the reduced space. Should be between 2 and min(n_samples-1, n_features).

metricstr, default=”euclidean”

Distance metric for UMAP. Supported metrics include ‘euclidean’, ‘manhattan’, ‘chebyshev’, ‘minkowski’, ‘cosine’, among others.

dataset{“main”, “train”, “test”}, default=”main”

Specifies which dataset partition to use for the analysis.

sample_sizeint, default=None

If set, randomly samples this many points from the dataset. Useful for large datasets.

categorical_encoding{“ordinal”, “onehot”}, default=”ordinal”

Method for encoding categorical variables:

• “ordinal”: Converts categories to integer values

• “onehot”: Creates binary columns for each category

random_stateint, default=0

Seed for random number generation, ensuring reproducibility.

Returns:

ValidationResult

A container object with the following components:

• key: “data_eda_umap”

• data: Name of the dataset used

• inputs: Dictionary of input parameters

• table: DataFrame containing the UMAP reduced dimensions

Examples

encode_categorical(features: str | Tuple = None, dataset: str = 'main', method: str = 'ordinal', target: str = None)

Encodes categorical features using either ordinal, one-hot, or target encoding methods.

This function transforms categorical variables into numerical format for machine learning models. It supports ordinal encoding (converting categories to integers), one-hot encoding (converting categories to binary columns), and target encoding (replacing categories with the mean of the target variable for each category). The encoding is fitted on the specified dataset and can be applied to new data using the transform method.

Parameters:

featuresstr or tuple, default=None

Feature names to be encoded. If None, all categorical features in the dataset will be automatically selected for encoding.

dataset{“main”, “train”, “test”}, default=”main”

Specifies which dataset partition to use for generating the binning boundaries.

method{“ordinal”, “onehot”, “target”}, default=”ordinal”

Encoding method to use:

• “ordinal”: Converts categories to integer values

• “onehot”: Creates binary columns for each category (minus one reference category)

• “target”: Replaces categories with the mean of the target variable for each category

targetstr, optional, default=None

The name of the target variable to use for target encoding. Required if method is “target”.

Returns:

ValidationResult

A container object with the following components:

• key: “data_preprocess_encoding”

• data: Name of the dataset used

• inputs: Dictionary of input parameters used

• value: Dictionary containing encoder configurations for each feature:

  • “fidx”: Feature index

  • “encoder”: Fitted encoder object

  • “feature_names_out”: List of output feature names

Examples

engineer_features()

Execute all queued feature engineering steps on the current data.

fe_bivector(features: str | Tuple = None, dataset: str = 'main', max_features: int = 300, selection: str = 'variance')

Generates grade-2 bivector features (pairwise feature products).

For each pair (i, j) with i < j, computes B_ij = X_i * X_j. This captures pairwise nonlinear interactions based on Geometric Algebra wedge products.

Parameters:

featuresstr or tuple, default=None

Features to use for bivector computation. If None, all numerical features are used.

dataset{“main”, “train”, “test”}, default=”main”

Dataset used to fit the transformer.

max_featuresint, default=300

Maximum number of bivector features to generate.

selection{“all”, “random”, “variance”}, default=”variance”

How to select pairs if max_features < C(n,2):

• “all”: use all pairs (ignore max_features)

• “random”: random selection

• “variance”: highest variance pairs

fe_blade_spectrum(features: str | Tuple = None, dataset: str = 'main', n_neighbors: int = 10)

Generates local blade energy spectrum features from k-NN neighborhood.

For each sample, queries k nearest neighbors in whitened space and computes grade 1/2/3 blade energies, normalized proportions, spectral slope, and dominant grade.

Parameters:

featuresstr or tuple, default=None

Features to use. If None, all numerical features are used.

dataset{“main”, “train”, “test”}, default=”main”

Dataset used to fit the transformer.

n_neighborsint, default=10

Number of nearest neighbors (k).

fe_density(features: str | Tuple = None, dataset: str = 'main', bandwidth: str | float = 'median', n_reference: int = 500, log_density: bool = True)

Generates RBF kernel density feature per sample.

Computes mean RBF similarity to a reference subset of training points, yielding a scalar density estimate per row.

Parameters:

featuresstr or tuple, default=None

Features to use. If None, all numerical features are used.

dataset{“main”, “train”, “test”}, default=”main”

Dataset used to fit the transformer.

bandwidthstr or float, default=”median”

If ‘median’, sigma = median(pairwise_dists) / sqrt(2*log(2)). If float, use directly as sigma.

n_referenceint, default=500

Number of reference points subsampled from training data.

log_densitybool, default=True

If True, return log(density + 1e-10), else return density directly.

fe_direct_rs(features: str | Tuple = None, dataset: str = 'main', n_estimators: int = 50, max_depth: int = 4, regularization: float = 1e-08)

Generates DirectRS stretch features from random forest leaf geometry.

Builds a weighted outer-product matrix from per-leaf centroids and variances, decomposes via eigendecomposition, and applies the resulting stretch matrix. Requires a target variable.

Parameters:

featuresstr or tuple, default=None

Features to use. If None, all numerical features are used.

dataset{“main”, “train”, “test”}, default=”main”

Dataset used to fit the transformer.

n_estimatorsint, default=50

Number of trees in the forest.

max_depthint, default=4

Maximum tree depth.

regularizationfloat, default=1e-8

Regularization added to diagonal of G before eigendecomposition.

fe_grade_ratio(features: str | Tuple = None, dataset: str = 'main', max_triplets: int = 50, regularization: float = 1e-08)

Generates grade-ratio shape feature measuring 3-way vs 2-way interaction complexity.

After whitening, computes ratio E3/E2 where E2 is summed pairwise blade energy and E3 is summed triplet blade energy. High ratio = dominant 3-body structure.

Parameters:

featuresstr or tuple, default=None

Features to use. If None, all numerical features are used.

dataset{“main”, “train”, “test”}, default=”main”

Dataset used to fit the transformer.

max_tripletsint, default=50

Maximum number of sampled triplets for E3 estimation.

regularizationfloat, default=1e-8

Regularization for the internal whitening step.

fe_kinematics(features: str | Tuple = None, dataset: str = 'main', n_timesteps: int = None, n_features_per_step: int = None)

Generates velocity, acceleration, and kinematic summary features from temporal sequences.

Extracts kinematic quantities from multi-step time series data: last velocity, last acceleration, mean speed, kinetic energy, and directional persistence.

Parameters:

featuresstr or tuple, default=None

Features to use (treated as flat temporal sequence). If None, all numerical features are used.

dataset{“main”, “train”, “test”}, default=”main”

Dataset used to fit the transformer.

n_timestepsint, default=None

Number of time steps. If None, inferred from data.

n_features_per_stepint, default=None

Features per time step. If None, inferred from data.

fe_multi_grade_shape(features: str | Tuple = None, dataset: str = 'main', grades: List[int] = None, n_bins: int = 5, regularization: float = 1e-08)

Generates multi-grade shape features (context distances using grades 2-4).

Computes Mahalanobis (grade 2), skewness-weighted (grade 3), and kurtosis-weighted (grade 4) distances to class or quantile-bin centroids. Requires a target variable.

Parameters:

featuresstr or tuple, default=None

Features to use. If None, all numerical features are used.

dataset{“main”, “train”, “test”}, default=”main”

Dataset used to fit the transformer.

gradeslist of int, default=None

Which grades to include (2, 3, and/or 4). Defaults to [2].

n_binsint, default=5

Number of bins for regression (uses quantiles).

regularizationfloat, default=1e-8

Regularization for covariance inversion.

fe_phase_encoder(features: str | Tuple = None, dataset: str = 'main', n_thresholds: int = 5, include_original: bool = False)

Generates smooth threshold-proximity encoding via phase differences.

For each feature, learns percentile thresholds and encodes each value as its cosine similarity to each threshold in normalized phase space. Avoids hard binning.

Parameters:

featuresstr or tuple, default=None

Features to encode. If None, all numerical features are used.

dataset{“main”, “train”, “test”}, default=”main”

Dataset used to fit the transformer.

n_thresholdsint, default=5

Number of threshold points per feature (evenly spaced percentiles).

include_originalbool, default=False

If True, prepend original features to output.

fe_rf_proximity(features: str | Tuple = None, dataset: str = 'main', n_estimators: int = 100, max_depth: int = 6, n_landmarks: int = 50)

Generates RF leaf co-occurrence proximity features via Nystrom approximation.

Trains a random forest and uses its leaf co-occurrence as a similarity kernel. The Nystrom method computes a fixed-width feature map from landmark points. Requires a target variable.

Parameters:

featuresstr or tuple, default=None

Features to use. If None, all numerical features are used.

dataset{“main”, “train”, “test”}, default=”main”

Dataset used to fit the transformer.

n_estimatorsint, default=100

Number of trees in the forest.

max_depthint, default=6

Maximum tree depth.

n_landmarksint, default=50

Number of Nystrom anchor points.

fe_shear(features: str | Tuple = None, dataset: str = 'main', threshold: float = 0.0)

Applies pairwise shear (decorrelation via residualization).

For each feature pair (i, j): X_i’ = X_i - beta_ij * X_j, where beta_ij = cov(X_i, X_j) / var(X_j).

Parameters:

featuresstr or tuple, default=None

Features to decorrelate. If None, all numerical features are used.

dataset{“main”, “train”, “test”}, default=”main”

Dataset used to fit the transformer.

thresholdfloat, default=0.0

Only apply shear if |beta_ij| > threshold.

fe_spectral_proximity(features: str | Tuple = None, dataset: str = 'main', n_estimators: int = 100, max_depth: int = 6, n_components: int = 10)

Generates spectral embedding features from RF proximity kernel.

Builds the RF proximity matrix, computes the normalized graph Laplacian, and extracts the smoothest eigenvectors as features. Uses Nystrom extension at transform time. Requires a target variable.

Parameters:

featuresstr or tuple, default=None

Features to use. If None, all numerical features are used.

dataset{“main”, “train”, “test”}, default=”main”

Dataset used to fit the transformer.

n_estimatorsint, default=100

Number of trees in the forest.

max_depthint, default=6

Maximum tree depth.

n_componentsint, default=10

Number of spectral components.

fe_wedge_bivector(features: str | Tuple = None, dataset: str = 'main', max_features: int = 300, selection: str = 'variance')

Generates wedge-weighted pairwise feature products.

Suppresses nearly-collinear pairs and amplifies orthogonal ones by weighting each product X_i * X_j by the column-level sin(angle) between features i and j.

Parameters:

featuresstr or tuple, default=None

Features to use. If None, all numerical features are used.

dataset{“main”, “train”, “test”}, default=”main”

Dataset used to fit the transformer.

max_featuresint, default=300

Maximum number of pairs to generate.

selection{“variance”, “random”}, default=”variance”

Pair selection method if max_features < C(d,2).

fe_whitening(features: str | Tuple = None, dataset: str = 'main', regularization: float = 1e-08)

Applies rotor whitening (GA Mahalanobis transformation).

Aligns data with principal axes using SVD-based whitening:

X’ = (X - mu) @ V @ diag(1/sigma)

Parameters:

featuresstr or tuple, default=None

Features to whiten. If None, all numerical features are used.

dataset{“main”, “train”, “test”}, default=”main”

Dataset used to fit the transformer.

regularizationfloat, default=1e-8

Regularization for variance computation.

feature_select_corr(dataset: str = 'train', method: str = 'pearson', threshold: float = 0.2, random_state: int = 0)

Selects features based on their correlation strength with the target variable.

Analyzes the relationship between features and the target variable using different correlation metrics:

• Pearson / Spearman / Kendall / XI correlation for numerical-numerical pairs;

• Theil’s U for categorical-categorical pairs;

• correlation ratio for numerical-categorical pairs.

Features with absolute correlation above the specified threshold are selected.

Parameters:

dataset{“main”, “train”, “test”}, default=”train”

Dataset partition to use for feature selection analysis.

method{“pearson”, “spearman”, “kendall”, “xicor”}, default=”pearson”

The algorithm for calculating the correlation when the variable and target are both numerical.

• “pearson”: Pearson correlation measures the linear relationship between two continuous variables. Its value ranges from −1 (perfect negative linear relationship) to 1 (perfect positive linear relationship), with 0 indicating no linear correlation. It is sensitive to linear relationships but not to nonlinear patterns.

• “spearman”: Spearman correlation assesses the strength and direction of a monotonic relationship between two variables, based on their ranks. It ranges from −1 to 1, where 1 indicates a perfect increasing monotonic relationship and −1 a perfect decreasing one. It is robust to outliers and can capture non-linear relationships.

• “kendall”: Kendall Tau measures the association between two ranked variables, focusing on the consistency of the order between them. Its value ranges from −1 (perfect discordance) to 1 (perfect concordance). It is particularly useful for ordinal data and is robust to outliers.

• “xicor”: XiCor detects both linear and nonlinear dependencies between continuous variables. It typically ranges from 0 (no dependence) to 1 (strong dependence), providing a more comprehensive view of relationships. Negative XI correlation does not have any innate significance, other than close to zero. See details in the paper [1].

[1] Chatterjee, S. (2021). A new coefficient of correlation. J. Amer. Statist. Assoc., 116, no. 536, 2009–2022.

thresholdfloat, default=0.2

Minimum absolute correlation value required for feature selection. Features with correlation above this threshold are considered important.

random_stateint, default=0

Random seed for reproducibility in calculations involving random sampling.

Returns:

ValidationResult

A container object with the following components:

• key: “data_fs_corr”

• data: Name of the dataset used

• inputs: Input parameters used for feature selection

• value: Dictionary containing:

  • “selected”: List of selected feature names

• table: DataFrame with feature names, importance scores, and selection status

• options: Dictionary of visualizations configuration for a horizontal bar plot of feature importance. Run results.plot() to show this plot.

Examples

feature_select_rcit(dataset: str = 'train', threshold: float = 1e-06, n_fourier: int = 25, n_fourier2: int = 5, n_forwards: int = 2, random_state: int = 0)

Performs feature selection using RCIT and FBEDk to identify important features based on conditional independence testing.

This method implements a two-stage feature selection process that combines Randomized Conditional Independence Test (RCIT) with Forward-Backward-Elimination with Early Dropping (FBEDk). It first performs forward selection to identify potentially important features, followed by backward elimination to remove redundant ones, using random Fourier features for non-parametric conditional independence testing.

Parameters:

dataset{“main”, “train”, “test”}, default=”train”

Dataset partition to use for feature selection.

thresholdfloat, default=1e-6

P-value threshold for feature significance; features with p-values below this are retained.

n_fourierint, default=25

Number of random Fourier features for conditioning set in RCIT test.

n_fourier2int, default=5

Number of random Fourier features for non-conditioning set in RCIT test.

n_forwardsint, default=2

Number of forward selection iterations in FBEDk algorithm.

random_stateint, default=0

Seed for random number generation to ensure reproducibility.

Returns:

ValidationResult

A container object with the following components:

• key: “data_fs_rcit”

• data: Name of the dataset used

• inputs: Dictionary of input parameters

• value: Dictionary containing:

  • “step_logs”: List of selection step information

  • “selected”: List of selected feature names

• table: DataFrame showing step-by-step feature selection status

• options: Dictionary of visualizations configuration for a feature selection heatmap plot of feature importance. Run results.plot() to show this plot.

Examples

feature_select_xgbpfi(dataset: str = 'train', n_repeats: int = 10, threshold: float = 0.1, random_state: int = 0)

Selects important features using XGBoost model and permutation importance analysis.

This function trains an XGBoost model and evaluates feature importance through permutation analysis. Features are selected based on their normalized importance scores exceeding a specified threshold. The analysis includes visualization of feature importance scores with a threshold line.

Parameters:

dataset{“main”, “train”, “test”}, default=”train”

Dataset partition to analyze. “main” uses full dataset, “train”/”test” use respective splits.

n_repeatsint, default=10

Number of permutation iterations per feature. Higher values give more stable importance scores but increase computation time.

thresholdfloat, default=0.1

Minimum normalized importance score for feature selection. Features with scores above this threshold are selected.

random_stateint, default=0

Random seed for reproducible permutation results.

Returns:

ValidationResult

A container object with the following components:

• key: “data_fs_xgbpfi”

• data: Name of the dataset used

• inputs: Input parameters used for the analysis

• value: Dictionary containing:

  • “selected”: List of selected feature names

• table: DataFrame with feature names, importance scores, and selection status

• options: Dictionary of visualizations configuration for a horizontal bar plot of feature importance. Run results.plot() to show this plot.

Examples

get_X_y_data(dataset: str = 'main')

Get the preprocessed data in the form of X, y, sample_weight.

Only active samples are returned. For extra data, no subsampling exists.

Parameters:

dataset{“main”, “train”, “test”}, default=”main”

The name of data split. It can also be other manually registered data split, if exists. Use the function get_data_list to check all available data splits.

Returns:

X: np.ndarray

The given dataset’s X, will be None if not available.

y: np.ndarray

The given dataset’s y, will be None if not available.

sample_weight: np.ndarray

The given dataset’s sample_weight, will be None if not available.

get_active_feature_idx()

Get the sample indices (active).

Returns:

np.ndarray of the active features’ index.

get_active_sample_idx(dataset: str = 'main')

Get the sample indices (active).

For extra datasets, it is assumed all samples are active.

Parameters:

dataset{“main”, “train”, “test”}, default=”main”

The name of data split. It can also be other manually registered data split, if exists. Use the function get_data_list to check all available data splits.

Returns:

np.ndarray of the given dataset’s sample index.

get_data(dataset: str = 'main', active_sample: bool = False, active_feature: bool = False)

Get the preprocessed data in the format of np.ndarray (all variables including X, y, sample_weight, etc.)

All sample index are returned, and it is not affected by the changes in active_sample_index. It is designed for data preprocessing steps. For modeling related steps, use get_X_y_data instead.

Parameters:

dataset{“main”, “train”, “test”}, default=”main”

The name of data split. It can also be other manually registered data split, if exists. Use the function get_data_list to check all available data splits.

active_samplebool, default=False

If True, will only return the active rows (samples).

active_featurebool, default=False

If True, will only return the active columns (features).

Returns:

np.ndarray of the given dataset’s preprocessed version.

get_data_list()

Get the available data split names.

Returns:

List of the all available dataset names.

get_extra_data_list()

Get the extra data split names.

Returns:

List of the extra dataset names.

get_feature_engineer()

Get the feature engineering aggregator.

get_preprocessor()

Get preprocessor.

get_protected_data(dataset: str = 'main')

Get the protected data in the format of np.ndarray (raw, as there is no preprocessed version of protected data).

Only active samples are returned. For extra data, no subsampling exists.

Parameters:

dataset{“main”, “train”, “test”}, default=”main”

The name of data split. It can also be other manually registered data split, if exists. Use the function get_data_list to check all available data splits.

Returns:

np.ndarray of the given dataset’s protected features, will be None if not available.

get_raw_data(dataset: str = 'main')

Get the raw data in the format of np.ndarray

Parameters:

dataset{“main”, “train”, “test”}, default=”main”

The name of data split. It can also be other manually registered data split, if exists. Use the function get_data_list to check all available data splits.

Returns:

np.ndarray of the given dataset’s raw version.

impute_missing(features: str | Tuple = None, dataset: str = 'main', method: str = 'mean', fill_value: int | float | str = None, missing_values: int | float | str = nan, special_values: int | float | str | list | tuple = None, add_indicators: bool = True)

Performs data imputation for missing values with optional indicator columns.

This function handles missing value imputation across multiple features using various strategies, with special consideration for categorical features. It can create indicator columns to track where imputation occurred and handles both numeric and categorical data appropriately.

Parameters:

featuresstr or tuple, default=None

Feature names to process. If None, all features in the dataset will be processed.

dataset{“main”, “train”, “test”}, default=”main”

The data to fit the imputer.

methodstr, default=’mean’

The imputation strategy. For categorical features, will use “most_frequent” by default.

• If “mean”, then replace missing values using the mean along each column. Can only be used with numeric data.

• If “median”, then replace missing values using the median along each column. Can only be used with numeric data.

• If “most_frequent”, then replace missing using the most frequent value along each column. Can be used with strings or numeric data. If there is more than one such value, only the smallest is returned.

• If “constant”, then replace missing values with fill_value. Can be used with strings or numeric data.

missing_valuesint, float, str, np.nan, None or pandas.NA, default=np.nan

The placeholder for the missing values. All occurrences of missing_values will be imputed. For pandas’ dataframes with nullable integer dtypes with missing values, missing_values can be set to either np.nan or pd.NA.

fill_valuestr or numerical value, default=None

When strategy == “constant”, fill_value is used to replace all occurrences of missing_values. For string or object data types, fill_value must be a string. If None, fill_value will be 0 when imputing numerical data and “missing_value” for strings or object data types.

add_indicatorsbool, default=True

If True, adds columns indicating where missing and special values were imputed.

special_valueslist, optional, default=None

A list of special values (e.g., [“999”, “888”]) to be treated as missing. Must be compatible with the data type of the columns.

Returns:

ValidationResult

A container object with the following components:

• key: “data_preprocess_imputing”

• data: Name of the dataset used

• inputs: Dictionary of input parameters

• value: Dictionary containing imputation configuration for each feature:

  • “fidx”: Feature index

  • “imputer”: Fitted IndicatorImputer instance

  • “feature_names_out”: List of output feature names including indicators

Examples

inverse_transform(data)

Convert preprocessed data to raw data.

Parameters:

datapd.DataFrame

The preprocessed data, e.g., ds.data

Returns:

pd.DataFrame of raw data

is_built_in()

Keep track if using built-in dataset or not.

Returns:

built-inbool

If it is using built-in dataset or not.

is_splitted()

Return True if data is already split, otherwise False.

Returns:

boolean indicator

list_registered_data(name: str = None)

Return the list of registered dataset in mlflow.

Parameters:

namestr, default=None

Name of data used for filtering. If None, will return all data.

Returns:

pd.Dataframe of registered data.

load(name: str)

Load built-in data.

Parameters:

namestr

Available built-in data includes

• “BikeSharing”

• “TaiwanCredit”

• “SimuCredit”

• “CaliforniaHousing”

load_csv(data_path: str, *args, **kwargs)

Load data from csv file.

Parameters:

data_pathstr

The path of csv data

args:

Positional arguments for pd.read_csv

kwargs:

Keyword arguments for pd.read_csv

load_dataframe(data: DataFrame | ndarray)

Load data from DataFrame or numpy ndarray.

Parameters:

datapandas DataFrame or numpy ndarray

The raw data.

load_dataframe_train_test(train: DataFrame | ndarray, test: DataFrame | ndarray)

Load data from DataFrame or numpy ndarray with given train test data.

Parameters:

trainpandas DataFrame or numpy ndarray

The raw data of training set.

testpandas DataFrame or numpy ndarray

The raw data of testing set.

load_preprocessing(path_or_buf: str | Path)

Load preprocessing steps from a saved file.

Parameters:

path_or_bufstr or Path

The path to the file containing saved preprocessing steps. The file should have been created using the save_preprocessing method.

Examples

>>> ds.load_preprocessing("path/to/preprocessing.pkl")

load_registered_data(name: str, version: int = None)

Load registered data.

1. Check if registered data exists

2. Get the latest version if no version is provided

3. Read csv file and meta info file

Parameters:

namestr, default=None

Name of data used for filtering. If None, will return all data.

versionint, default=None

The version of data.

load_spark(data_path: str)

Load data from spark file.

Parameters:

data_pathstr

The path of spark data

preprocess()

Preprocess the internal raw data and extra data based on existing settings.

register(name: str = None, description: str | None = None, tags: Dict[str, Any] | None = None, override: bool = False)

Register the dataset into mlflow.

Parameters:

namestr, default=None

Register name. If None, will use dataset name.

descriptionstr, default=None

A str describing the data.

tagsdict, default=None

A dictionary containing the tags of this data.

overridebool, default=False

if the register name is registered before, it will not be registered

reset_feature_engineering()

Remove all previous feature engineering steps.

reset_preprocess()

Remove all previous preprocess steps.

save(path_or_buf: str | Path, run_id: str)

Save the raw data to csv file.

Parameters:

path_or_bufstr or Path

The path of the file.

run_idstr

The run id of mlflow.

save_meta_info(path_or_buf: str | Path, run_id: str)

Save the meta_info to file.

Parameters:

path_or_bufstr or Path

The path of the file.

run_idstr

The run id of mlflow.

save_preprocessing(path_or_buf: str | Path, run_id: str)

Save the preprocessing steps to file.

Parameters:

path_or_bufstr or Path

The path of the file.

run_idstr

The run id of mlflow.

scale_numerical(features: str | Tuple = None, dataset: str = 'main', method: str = 'minmax', minmax_range: tuple | None = (0, 1), n_quantiles: int = 1000)

Scales numerical features using various scaling methods.

Applies the specified scaling transformation to selected numerical features in the dataset. The scaling parameters are computed based on the specified dataset and can be used to transform new data consistently.

Parameters:

featuresstr or tuple, default=None

Features to be scaled. If None, all numerical features in the dataset will be scaled.

dataset{“main”, “train”, “test”}, default=”main”

Dataset used to compute scaling parameters (e.g., mean, std, min, max).

method{“standardize”, “minmax”, “quantile”, “log1p”, “square”}, default=”minmax”

Scaling method to apply:

• “standardize”: Centers data to zero mean and unit variance

• “minmax”: Scales data to a specific range

• “quantile”: Transforms using quantile information for robust scaling

• “log1p”: Applies natural logarithm plus one transformation

• “square”: Applies square transformation

minmax_rangetuple, default=(0, 1)

Target range for minmax scaling, specified as (min, max).

n_quantilesint, default=1000

Number of quantiles for quantile transformation. Limited by sample size.

Returns:

ValidationResult

A container object with the following components:

• key: “data_preprocess_scaling”

• data: Name of the dataset used

• inputs: Input parameters used for scaling

• value: Dictionary containing scaling configuration (<feature_name>, item) pairs, each item is a dictionary contains following:

  • “fidx”: Feature index

  • “scalers”: Function for scaling

  • “feature_names_out”: List of output feature names

Examples

set_active_features(features=None)

Set active features.

The existing active and inactive features will be overridden.

Parameters:

featureslist or tuple of str, default=None

The name of active features. If None, all features will be set active.

set_active_samples(dataset: str = 'main', sample_idx: ndarray | DataFrame = None)

Set some samples as active, and the rest samples are auto set to be inactive.

The existing active / inactive samples will be overridden per “main”, “train”, “test” sets. It can be used for e.g., outlier removal.

Parameters:

dataset{“main”, “train”, “test”}, default=”main”

The data set to be chosen.

sample_idxnp.ndarray or pd.DataFrame, default=None

The sample index to be removed. It corresponds to the sample index in the selected dataset. If None, all samples will be reset to be active.

set_feature_type(feature: str, feature_type: str)

Update the type of selected feature.

Parameters:

featurestr

The name of the selected feature.

feature_type{“numerical”, “categorical”, “mixed”, “date”}

The type of this feature.

set_inactive_features(features=None)

Set inactive features.

The existing active and inactive features will be overridden.

Parameters:

featureslist or tuple of str, default=None

The name of inactive features. If None, all features will be set active.

set_inactive_samples(dataset: str = 'main', sample_idx: ndarray = None)

Set some samples as inactive, and the rest samples are auto set to be active.

The existing active / inactive samples will be overridden per “main”, “train”, “test” sets. It can be used for e.g., outlier removal.

Parameters:

dataset{“main”, “train”, “test”}, default=”main”

The data set to be chosen.

sample_idxnp.ndarray, default=None

The sample index to be removed. It corresponds to the sample index in the selected dataset. If None, all samples will be treated as active.

set_prediction(feature: str)

Set the column that will be used as prediction.

Parameters:

featurestr

The name of the sample weight feature.

set_prediction_proba(feature: str)

Set the column that will be used as prediction probability.

Parameters:

featurestr

The name of the sample weight feature.

set_protected_data(data: DataFrame | ndarray)

Set protected features, e.g., demographic features.

The data should have the same length as the raw data.

Parameters:

datapd.DataFrame or np.ndarray

The protected data.

set_protected_extra_data(name: str, data: DataFrame | ndarray)

Set protected features, e.g., demographic features in extra data.

The data should have the same length as the extra data.

Parameters:

namestr

The name of the data split.

datapd.DataFrame or np.ndarray

The protected data.

set_random_split(test_ratio: float = 0.2, shuffle: bool = True, random_state: int = 0)

Set random train-test split on active samples.

Parameters:

test_ratiofloat, default=0.2

The percentage of test samples.

shufflebool, default=True

Whether to shuffle data before splitting.

random_stateint, default=0

The seed for controlling randomness.

set_raw_extra_data(name: str, data: DataFrame | ndarray)

Add a new group of data to Dataset.

This data split will be treated like “train” or “test”, and can be used in downstream tests.

Returns:

namestr

The name of this data group, e.g., “oot” (stands for out of time data).

datapd.DataFrame or np.ndarray

The data should have the same format as the raw data.

set_sample_weight(feature: str)

Set the column that will be used as sample weight.

Parameters:

featurestr

The name of the sample weight feature.

set_target(feature: str)

Set the target feature.

The task type will be automatically inferred from the type of the target feature. You may manually modify the task type using the function ‘self.set_task_type’.

Parameters:

featurestr

The name of the target feature.

set_task_type(task_type: str)

Set the task type.

Parameters:

task_type{“Regression”, “Classification”}

Supported tasks include regression and binary classification.

set_test_idx(test_idx)

Set testing indices.

Parameters:

test_idxnumpy ndarray of int

The indices of testing samples.

set_train_idx(train_idx)

Set training indices.

Parameters:

train_idxnumpy ndarray of int

The indices of training samples.

subsample_random(dataset: str = 'main', sample_size: [<class 'int'>, <class 'float'>] = 0.2, shuffle: bool = True, stratify: str = None, random_state: int = 0)

Subsample data randomly.

This function performs random subsampling on the specified dataset, allowing for options such as shuffling and stratification.

Parameters:

dataset{“main”, “train”, “test”}, default=”main”

Specifies which dataset to use for subsampling. Options include “main”, “train”, or “test”.

sample_sizeint or float, default = 10

If it is a float, should be between 0.0 and 1.0 and represent the proportion of the dataset to include in the test split. If it is an int, represents the absolute number of test samples. If None, the value is set to the complement of the train size. If train_size is also None, it will be set to 0.25.

shufflebool, default=True

Indicates whether to shuffle the data before subsampling. If set to False, stratification cannot be applied.

stratifystr, default=None

The name of the feature to use for stratified sampling. If provided, the data will be split in a stratified manner based on this feature.

random_stateint, default=0

The seed used by the random number generator, ensuring reproducibility of the results.

Returns:

ValidationResult

A container object with the following components:

• key: “subsample_random”

• data: Name of the dataset used

• inputs: Input parameters

• value: Dictionary containing:

  • “sample_idx”: Indices of the sampled data points

Examples

summary(dataset: str = 'main')

Generates comprehensive descriptive statistics and analysis of the dataset.

This function performs a detailed analysis of the dataset, computing various statistical measures and summaries for different types of features (numerical, categorical, and mixed). It analyzes the overall dataset structure and provides detailed statistics for each feature based on its type.

Parameters:

dataset{“main”, “train”, “test”}, default=”main”

Specifies which dataset partition to analyze. Use “main” for the complete dataset, “train” for training data, or “test” for test data.

Returns:

ValidationResult

A container object with the following components:

• key: “data_summary”

• data: Name of the analyzed dataset

• inputs: Input parameters used for the analysis

• table: Dictionary containing DataFrames with:

  • “summary”: Overall dataset statistics

  • “numerical”: Statistics for numerical features

  • “categorical”: Statistics for categorical features

  • “mixed”: Statistics for mixed-type features

• value: Dictionary containing raw analysis results:

  • “features”: Detailed statistics for each feature

  • “summary”: Overall dataset metrics

Examples

to_df(raw_data: bool = False)

Return the source data using pandas DataFrame.

Parameters:

raw_databool, default=False

If true, will return the raw data. Otherwise, will return the preprocessed version of data.

Returns:

datapandas DataFrame

The raw data.

transform(data)

Convert raw data to preprocessed data.

Parameters:

datapd.DataFrame

The raw data, e.g., ds.raw_data

Returns:

pd.DataFrame of preprocessed data

transform_features(data)

Apply fitted feature engineering to new data.

Parameters:

datapd.DataFrame

The data to transform.

Returns:

pd.DataFrame with engineered features appended.

property active_sample_idx

Get the active sample indices of “main” data

property all_feature_names

Get the list of all column names of the data. (preprocessed data)

property all_feature_types

Get the list of all column types of the data, including “categorical” and “numerical”. (preprocessed data)

property data

Return the preprocessed version of data as pd.DataFrame.

property extra_data

Return the preprocessed version of data as pd.DataFrame.

property feature_indices

Get the list of selected feature indices (only X).

property feature_names

Get the list of selected feature names (only X).

property feature_names_categorical

Get the list of selected categorical feature names (only X).

property feature_names_mixed

Get the list of selected mixed type feature names (only X).

property feature_names_numerical

Get the list of selected numerical feature names (only X).

property feature_types

Get the list of selected feature types (only X).

property inactive_sample_idx

Get the inactive sample indices of “main” data.

property inactive_sample_idx_test

Get the inactive sample indices of “test” data.

property inactive_sample_idx_train

Get the inactive sample indices of “train” data.

property main_idx

Get the main set indices (active).

property meta_info

Return the meta information of this data.

property n_features

Get the number of selected features (only X).

property name

Return the name of this dataset.

property prediction_name

Get the prediction column name.

property prediction_proba_name

Get the prediction probability column name.

property protected_data

Return the protected raw data as pd.DataFrame.

property protected_extra_data

Return the protected raw extra data as pd.DataFrame.

property protected_feature_names

Get the list of all column names of the raw data.

property random_state

Get the random state of data splitting.

property raw_data

Return the raw data as pd.DataFrame.

property raw_extra_data

Return the raw data as pd.DataFrame.

property raw_test_idx

Get the testing indices (raw).

property raw_train_idx

Get the training indices (raw).

property sample_weight

Get the preprocessed version of data of selected sample weight.

property sample_weight_index

Get the column index of sample weight.

property sample_weight_name

Get the selected sample weight column name.

property shape

Return the shape of this data.

property target_feature_index

Get the index of the selected target feature (only y).

property target_feature_name

Get the selected target feature name (only y).

property target_feature_type

Get the selected target feature type (only y).

property task_type

Get the task type, including “Regression” and “Classification”.

property test_idx

Get the testing set indices (active).

property test_sample_weight

Get the testing sample_weight (active and preprocessed).

property test_x

Get the testing x.

property test_y

Get the testing y (active and preprocessed).

property train_idx

Get the training set indices (active).

property train_sample_weight

Get the training sample_weight (active and preprocessed).

property train_x

Get the training x (active and preprocessed).

property train_y

Get the training y.

property version

Return the version of this data.

property x

Get the preprocessed version of data of selected features (only X).

property y

Get the preprocessed version of data of selected target feature (only y).