DataSet

The DataSet class (exposed as modeva.DataSet) handles data loading, exploration, preprocessing, feature selection, outlier detection, splitting, drift and sampling.

LocalDataSet

__init__(name: str='dataset', experiment_name: str='default')

summary(dataset: str='main')

eda_1d(feature: str, dataset: str='main', plot_type: str='density', bins: int=10, sample_size: int=None, random_state: int=0)

eda_2d(feature_x: str, feature_y: str, feature_color: str=None, dataset: str='main', sample_size: int=None, smoother_order: int=None, random_state: int=0)

eda_3d(feature_x: str, feature_y: str, feature_z: str, feature_color: str=None, dataset: str='main', sample_size: int=1000, random_state: int=0)

eda_correlation(features: Union[Tuple, List]=None, dataset: str='main', method: str='pearson', sample_size: int=None, random_state: int=0)

eda_pca(features: Union[Tuple, List]=None, n_components: int=None, dataset: str='main', sample_size: int=None, categorical_encoding: str='ordinal', random_state: int=0)

eda_umap(features: Union[Tuple, List]=None, n_neighbors: int=15, n_components: int=2, metric: str='euclidean', dataset: str='main', sample_size: int=None, categorical_encoding: str='ordinal', random_state: int=0)

impute_missing(features: Union[str, Tuple]=None, dataset: str='main', method: str='mean', fill_value: Union[int, float, str]=None, missing_values: Union[int, float, str]=np.nan, special_values: Union[int, float, str, list, tuple]=None, add_indicators: bool=True)

scale_numerical(features: Union[str, Tuple]=None, dataset: str='main', method: str='minmax', minmax_range: Optional[tuple]=(0, 1), n_quantiles: int=1000)

bin_numerical(features: Union[str, Tuple]=None, dataset: str='main', method: str='uniform', bins: Union[int, Dict]=10)

encode_categorical(features: Union[str, Tuple]=None, dataset: str='main', method: str='ordinal', target: str=None)

get_preprocessor()

Get preprocessor.

reset_preprocess()

Remove all previous preprocess steps.

preprocess()

Preprocess the internal raw data and extra data based on existing settings.

transform(data)

Convert raw data to preprocessed data.

Parameters

data : pd.DataFrame
The raw data, e.g., ds.raw_data

Returns

pd.DataFrame of preprocessed data

inverse_transform(data)

Convert preprocessed data to raw data.

Parameters

data : pd.DataFrame
The preprocessed data, e.g., ds.data

Returns

pd.DataFrame of raw data

fe_bivector(features: Union[str, Tuple]=None, dataset: str='main', max_features: int=300, selection: str='variance')

fe_wedge_bivector(features: Union[str, Tuple]=None, dataset: str='main', max_features: int=300, selection: str='variance')

fe_whitening(features: Union[str, Tuple]=None, dataset: str='main', regularization: float=1e-08)

fe_shear(features: Union[str, Tuple]=None, dataset: str='main', threshold: float=0.0)

fe_grade_ratio(features: Union[str, Tuple]=None, dataset: str='main', max_triplets: int=50, regularization: float=1e-08)

fe_multi_grade_shape(features: Union[str, Tuple]=None, dataset: str='main', grades: List[int]=None, n_bins: int=5, regularization: float=1e-08)

fe_density(features: Union[str, Tuple]=None, dataset: str='main', bandwidth: Union[str, float]='median', n_reference: int=500, log_density: bool=True)

fe_blade_spectrum(features: Union[str, Tuple]=None, dataset: str='main', n_neighbors: int=10)

fe_rf_proximity(features: Union[str, Tuple]=None, dataset: str='main', n_estimators: int=100, max_depth: int=6, n_landmarks: int=50)

fe_spectral_proximity(features: Union[str, Tuple]=None, dataset: str='main', n_estimators: int=100, max_depth: int=6, n_components: int=10)

fe_direct_rs(features: Union[str, Tuple]=None, dataset: str='main', n_estimators: int=50, max_depth: int=4, regularization: float=1e-08)

fe_phase_encoder(features: Union[str, Tuple]=None, dataset: str='main', n_thresholds: int=5, include_original: bool=False)

fe_kinematics(features: Union[str, Tuple]=None, dataset: str='main', n_timesteps: int=None, n_features_per_step: int=None)

get_feature_engineer()

Get the feature engineering aggregator.

reset_feature_engineering()

Remove all previous feature engineering steps.

engineer_features()

Execute all queued feature engineering steps on the current data.

transform_features(data)

Apply fitted feature engineering to new data.

Parameters

data : pd.DataFrame
The data to transform.

Returns

pd.DataFrame with engineered features appended.

feature_select_corr(dataset: str='train', method: str='pearson', threshold: float=0.2, random_state: int=0)

feature_select_xgbpfi(dataset: str='train', n_repeats: int=10, threshold: float=0.1, random_state: int=0)

feature_select_rcit(dataset: str='train', threshold: float=1e-06, n_fourier: int=25, n_fourier2: int=5, n_forwards: int=2, random_state: int=0)

detect_outlier_pca(dataset: str='main', threshold: float=0.99, cumulative_variance_threshold: float=0.9, method: str='mahalanobis', sparse_pca: bool=False, alpha: float=1.0, random_state: int=0)

detect_outlier_isolation_forest(dataset: str='main', threshold: float=0.99, n_estimators: int=100)

detect_outlier_cblof(dataset: str='main', threshold: float=0.99, method: str='kmeans', n_clusters: int=10, cluster_threshold: float=0.1, use_weights: bool=False, random_state: int=0)

subsample_random(dataset: str='main', sample_size: [int, float]=0.2, shuffle: bool=True, stratify: str=None, random_state: int=0)

data_drift_test(dataset1: str='train', dataset2: str='test', sample_idx1: Union[Tuple[int], np.ndarray]=None, sample_idx2: Union[Tuple[int], np.ndarray]=None, name1: str=None, name2: str=None, distance_metric: str='PSI', psi_method: str='uniform', psi_bins: int=10)

save_preprocessing(path_or_buf: Union[str, Path], run_id: str)

Save the preprocessing steps to file.

Parameters

path_or_buf : str or Path
The path of the file.

run_id : str
The run id of mlflow.

load_preprocessing(path_or_buf: Union[str, Path])

Load preprocessing steps from a saved file.

Parameters

path_or_buf : str or Path
The path to the file containing saved preprocessing steps. The file should have been created using the save_preprocessing method.

Examples

>>> ds.load_preprocessing(“path/to/preprocessing.pkl”)

is_built_in()

Keep track if using built-in dataset or not.

Returns

built-in : bool
If it is using built-in dataset or not.

load(name: str)

Load built-in data.

Parameters

name : str
Available built-in data includes

• “BikeSharing”
• “TaiwanCredit”
• “SimuCredit”
• “CaliforniaHousing”

load_csv(data_path: str, *args, **kwargs)

Load data from csv file.

Parameters

data_path : str
The path of csv data

args:
Positional arguments for pd.read_csv

kwargs:
Keyword arguments for pd.read_csv

load_spark(data_path: str)

Load data from spark file.

Parameters

data_path : str
The path of spark data

load_dataframe(data: Union[pd.DataFrame, np.ndarray])

Load data from DataFrame or numpy ndarray.

Parameters

data : pandas DataFrame or numpy ndarray
The raw data.

load_dataframe_train_test(train: Union[pd.DataFrame, np.ndarray], test: Union[pd.DataFrame, np.ndarray])

Load data from DataFrame or numpy ndarray with given train test data.

Parameters

train : pandas DataFrame or numpy ndarray
The raw data of training set.

test : pandas DataFrame or numpy ndarray
The raw data of testing set.

to_df(raw_data: bool=False)

Return the source data using pandas DataFrame.

Parameters

raw_data : bool, default=False
If true, will return the raw data. Otherwise, will return the preprocessed version of data.

Returns

data : pandas DataFrame
The raw data.

set_raw_extra_data(name: str, data: Union[pd.DataFrame, np.ndarray])

Add a new group of data to Dataset.

This data split will be treated like “train” or “test”, and can be used in downstream tests.

Returns

name : str
The name of this data group, e.g., “oot” (stands for out of time data).

data : pd.DataFrame or np.ndarray
The data should have the same format as the raw data.

set_protected_data(data: Union[pd.DataFrame, np.ndarray])

Set protected features, e.g., demographic features.

The data should have the same length as the raw data.

Parameters

data : pd.DataFrame or np.ndarray
The protected data.

set_protected_extra_data(name: str, data: Union[pd.DataFrame, np.ndarray])

Set protected features, e.g., demographic features in extra data.

The data should have the same length as the extra data.

Parameters

name : str
The name of the data split.

data : pd.DataFrame or np.ndarray
The protected data.

delete_extra_data(name: str)

Delete a group of data.

Protected features will also be deleted if specified.

Parameters

name : str
The name of this data group, e.g., “oot” (stands for out of time data).

name()

Return the name of this dataset.

raw_data()

Return the raw data as pd.DataFrame.

data()

Return the preprocessed version of data as pd.DataFrame.

raw_extra_data()

Return the raw data as pd.DataFrame.

extra_data()

Return the preprocessed version of data as pd.DataFrame.

protected_data()

Return the protected raw data as pd.DataFrame.

protected_extra_data()

Return the protected raw extra data as pd.DataFrame.

meta_info()

Return the meta information of this data.

version()

Return the version of this data.

shape()

Return the shape of this data.

all_feature_names()

Get the list of all column names of the data. (preprocessed data)

all_feature_types()

Get the list of all column types of the data, including “categorical” and “numerical”. (preprocessed data)

protected_feature_names()

Get the list of all column names of the raw data.

feature_indices()

Get the list of selected feature indices (only X).

feature_names()

Get the list of selected feature names (only X).

feature_types()

Get the list of selected feature types (only X).

feature_names_numerical()

Get the list of selected numerical feature names (only X).

feature_names_categorical()

Get the list of selected categorical feature names (only X).

feature_names_mixed()

Get the list of selected mixed type feature names (only X).

n_features()

Get the number of selected features (only X).

target_feature_index()

Get the index of the selected target feature (only y).

target_feature_name()

Get the selected target feature name (only y).

target_feature_type()

Get the selected target feature type (only y).

sample_weight_index()

Get the column index of sample weight.

sample_weight_name()

Get the selected sample weight column name.

prediction_name()

Get the prediction column name.

prediction_proba_name()

Get the prediction probability column name.

x()

Get the preprocessed version of data of selected features (only X).

y()

Get the preprocessed version of data of selected target feature (only y).

sample_weight()

Get the preprocessed version of data of selected sample weight.

set_feature_type(feature: str, feature_type: str)

Update the type of selected feature.

Parameters

feature : str
The name of the selected feature.

feature_type : {“numerical”, “categorical”, “mixed”, “date”}
The type of this feature.

set_active_features(features=None)

Set active features.

The existing active and inactive features will be overridden.

Parameters

features : list or tuple of str, default=None
The name of active features. If None, all features will be set active.

set_inactive_features(features=None)

Set inactive features.

The existing active and inactive features will be overridden.

Parameters

features : list or tuple of str, default=None
The name of inactive features. If None, all features will be set active.

active_sample_idx()

Get the active sample indices of “main” data

inactive_sample_idx()

Get the inactive sample indices of “main” data.

inactive_sample_idx_train()

Get the inactive sample indices of “train” data.

inactive_sample_idx_test()

Get the inactive sample indices of “test” data.

set_active_samples(dataset: str='main', sample_idx: Union[np.ndarray, pd.DataFrame]=None)

Set some samples as active, and the rest samples are auto set to be inactive.

The existing active / inactive samples will be overridden per “main”, “train”, “test” sets. It can be used for e.g., outlier removal.

Parameters

dataset : {“main”, “train”, “test”}, default=“main”
The data set to be chosen.

sample_idx : np.ndarray or pd.DataFrame, default=None
The sample index to be removed. It corresponds to the sample index in the selected dataset. If None, all samples will be reset to be active.

set_inactive_samples(dataset: str='main', sample_idx: np.ndarray=None)

Set some samples as inactive, and the rest samples are auto set to be active.

The existing active / inactive samples will be overridden per “main”, “train”, “test” sets. It can be used for e.g., outlier removal.

Parameters

dataset : {“main”, “train”, “test”}, default=“main”
The data set to be chosen.

sample_idx : np.ndarray, default=None
The sample index to be removed. It corresponds to the sample index in the selected dataset. If None, all samples will be treated as active.

set_target(feature: str)

Set the target feature.

The task type will be automatically inferred from the type of the target feature. You may manually modify the task type using the function ‘self.set_task_type’.

Parameters

feature : str
The name of the target feature.

task_type()

Get the task type, including “Regression” and “Classification”.

set_task_type(task_type: str)

Set the task type.

Parameters

task_type : {“Regression”, “Classification”}
Supported tasks include regression and binary classification.

set_sample_weight(feature: str)

Set the column that will be used as sample weight.

Parameters

feature : str
The name of the sample weight feature.

set_prediction(feature: str)

Set the column that will be used as prediction.

Parameters

feature : str
The name of the sample weight feature.

set_prediction_proba(feature: str)

Set the column that will be used as prediction probability.

Parameters

feature : str
The name of the sample weight feature.

random_state()

Get the random state of data splitting.

train_x()

Get the training x (active and preprocessed).

train_y()

Get the training y.

train_sample_weight()

Get the training sample_weight (active and preprocessed).

test_x()

Get the testing x.

test_y()

Get the testing y (active and preprocessed).

test_sample_weight()

Get the testing sample_weight (active and preprocessed).

raw_train_idx()

Get the training indices (raw).

raw_test_idx()

Get the testing indices (raw).

main_idx()

Get the main set indices (active).

train_idx()

Get the training set indices (active).

test_idx()

Get the testing set indices (active).

set_train_idx(train_idx)

Set training indices.

Parameters

train_idx : numpy ndarray of int
The indices of training samples.

set_test_idx(test_idx)

Set testing indices.

Parameters

test_idx : numpy ndarray of int
The indices of testing samples.

set_random_split(test_ratio: float=0.2, shuffle: bool=True, random_state: int=0)

Set random train-test split on active samples.

Parameters

test_ratio : float, default=0.2
The percentage of test samples.

shuffle : bool, default=True
Whether to shuffle data before splitting.

random_state : int, default=0
The seed for controlling randomness.

is_splitted()

Return True if data is already split, otherwise False.

Returns

boolean indicator

get_extra_data_list()

Get the extra data split names.

Returns

List of the extra dataset names.

get_data_list()

Get the available data split names.

Returns

List of the all available dataset names.

get_active_sample_idx(dataset: str='main')

Get the sample indices (active).

For extra datasets, it is assumed all samples are active.

Parameters

dataset : {“main”, “train”, “test”}, default=“main”
The name of data split. It can also be other manually registered data split, if exists. Use the function get_data_list to check all available data splits.

Returns

np.ndarray of the given dataset’s sample index.

get_active_feature_idx()

Get the sample indices (active).

Returns

np.ndarray of the active features’ index.

get_raw_data(dataset: str='main')

Get the raw data in the format of np.ndarray

Parameters

dataset : {“main”, “train”, “test”}, default=“main”
The name of data split. It can also be other manually registered data split, if exists. Use the function get_data_list to check all available data splits.

Returns

np.ndarray of the given dataset’s raw version.

get_data(dataset: str='main', active_sample: bool=False, active_feature: bool=False)

Get the preprocessed data in the format of np.ndarray (all variables including X, y, sample_weight, etc.)

All sample index are returned, and it is not affected by the changes in active_sample_index. It is designed for data preprocessing steps. For modeling related steps, use get_X_y_data instead.

Parameters

dataset : {“main”, “train”, “test”}, default=“main”
The name of data split. It can also be other manually registered data split, if exists. Use the function get_data_list to check all available data splits.

active_sample : bool, default=False
If True, will only return the active rows (samples).

active_feature : bool, default=False
If True, will only return the active columns (features).

Returns

np.ndarray of the given dataset’s preprocessed version.

get_protected_data(dataset: str='main')

Get the protected data in the format of np.ndarray (raw, as there is no preprocessed version of protected data).

Only active samples are returned. For extra data, no subsampling exists.

Parameters

dataset : {“main”, “train”, “test”}, default=“main”
The name of data split. It can also be other manually registered data split, if exists. Use the function get_data_list to check all available data splits.

Returns

np.ndarray of the given dataset’s protected features, will be None if not available.

get_X_y_data(dataset: str='main')

Get the preprocessed data in the form of X, y, sample_weight.

Only active samples are returned. For extra data, no subsampling exists.

Parameters

dataset : {“main”, “train”, “test”}, default=“main”
The name of data split. It can also be other manually registered data split, if exists. Use the function get_data_list to check all available data splits.

Returns

X: np.ndarray
The given dataset’s X, will be None if not available.

y: np.ndarray
The given dataset’s y, will be None if not available.

sample_weight: np.ndarray
The given dataset’s sample_weight, will be None if not available.

save(path_or_buf: Union[str, Path], run_id: str)

Save the raw data to csv file.

Parameters

path_or_buf : str or Path
The path of the file.

run_id : str
The run id of mlflow.

save_meta_info(path_or_buf: Union[str, Path], run_id: str)

Save the meta_info to file.

Parameters

path_or_buf : str or Path
The path of the file.

run_id : str
The run id of mlflow.

register(name: str=None, description: Optional[str]=None, tags: Optional[Dict[str, Any]]=None, override: bool=False)

Register the dataset into mlflow.

Parameters

name : str, default=None
Register name. If None, will use dataset name.

description : str, default=None
A str describing the data.

tags : dict, default=None
A dictionary containing the tags of this data.

override : bool, default=False
if the register name is registered before, it will not be registered

list_registered_data(name: str=None)

Return the list of registered dataset in mlflow.

Parameters

name : str, default=None
Name of data used for filtering. If None, will return all data.

Returns

pd.Dataframe of registered data.

load_registered_data(name: str, version: int=None)

Load registered data.

1. Check if registered data exists
2. Get the latest version if no version is provided
3. Read csv file and meta info file

Parameters

name : str, default=None
Name of data used for filtering. If None, will return all data.

version : int, default=None
The version of data.

delete_registered_data(name: str)

Delete a registered dataset from MLflow.

Parameters

name : str
Name of the registered dataset to delete.

Raises

MlflowException
If the dataset doesn’t exist or there are permission issues.