Test Suite

The TestSuite class (exposed as modeva.TestSuite) provides post-hoc explanation, inherent interpretation, diagnostics and model comparison.

class modeva.testsuite.local_testsuite.LocalTestSuite(dataset: LocalDataSet = None, model: ModelBase = None, models: List[ModelBase] = None, name: str = 'testsuite')

A comprehensive model evaluation and analysis toolkit that provides methods for explaining, diagnosing, comparing, and interpreting machine learning models.

This class serves as the main interface for model evaluation, offering capabilities to: - Explain model behavior using various techniques (PFI, SHAP, LIME, etc.) - Diagnose model performance (accuracy, reliability, robustness, fairness, etc.) - Compare multiple models across different metrics - Interpret model decisions and feature importance - Register and manage test results using MLflow

Parameters:

datasetLocalDataSet, optional

The dataset object to be used for evaluation

modelModelBase, optional

The primary model to be evaluated

modelsList[ModelBase], optional

List of models for comparison purposes

namestr, default=”testsuite”

Name of the testsuite, used for MLflow experiment tracking

Examples

>>> # Create a testsuite for single model evaluation
>>> testsuite = LocalTestSuite(dataset=my_dataset, model=my_model, name="model_evaluation")
>>> 
>>> # Run accuracy diagnosis
>>> result = testsuite.diagnose_accuracy_table()
>>> 
>>> # Compare multiple models
>>> comparison_sheet = LocalTestSuite(dataset=my_dataset, models=[model1, model2])
>>> comparison = comparison_sheet.compare_accuracy_table()

classmethod list()

List all the experiments saved in database.

Returns:

pd.DataFrame

A table showing the experiments details in database.

compare_accuracy_table(train_dataset: str = 'train', test_dataset: str = 'test', metric: str | Tuple = None)

Compare predictive performance metrics across multiple models.

This function evaluates and compares the predictive performance of multiple models using specified metrics on both training and test datasets.

Parameters:

train_datasetstr, default=”train”

Specifies the training dataset to evaluate. Options: “main”, “train”, or “test”

test_datasetstr, default=”test”

Specifies the test dataset to evaluate. Options: “main”, “train”, or “test”

metricstr or tuple of str, default=None

Performance metric(s) to calculate. If None: - For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier” - For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

Returns:

ValidationResult

A container object with the following components:

• key: “compare_accuracy_table”

• data: Name of the dataset used

• model: List of model names being compared

• inputs: Input parameters used for the comparison

• value: Dictionary of (“<model_name>”, item), each item is also a nested dictionary with (“<metric_name>”, subitem), where each subitem is also a dictionary with:

  • “<train_dataset>”: The metric value of training dataset.

  • “<test_dataset>”: The metric value of testing dataset.

  • “GAP”: The performance gap is calculated as (test_score - train_score).

• table: Pandas DataFrame containing detailed performance metrics

• value: Dictionary of (“<model_name>”, item), each item is also a nested dictionary with (“<metric_name>”, subitem), where each subitem is also a dictionary with:

  • “<train_dataset>”: The metric value of training dataset.

  • “<test_dataset>”: The metric value of testing dataset.

  • “GAP”: The performance gap is calculated as (test_score - train_score).

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

  • “<metric_name>”: a bar plot where x-axis is the model names, and y-axis is performance metric

Examples

compare_fairness(group_config, favorable_label: int = 1, dataset: str = 'test', metric: str = None, threshold: float | int = None)

Compares fairness metrics across multiple models.

This function evaluates and compares fairness metrics for various models based on the provided group configurations, allowing for a comprehensive analysis of model performance across different demographic groups.

Parameters:

group_configdict

Configuration defining protected and reference groups. Each key is a custom group name, and each value is a dictionary with group definitions. Supports three formats:

favorable_label{0, 1}, default=1

For classification: The preferred class label. For regression: 1 means larger predictions are preferred, 0 means smaller predictions are preferred.

dataset{“main”, “train”, “test”}, 1. For numerical features:

{
    "feature": str,           # Feature name
    "protected": {            # Protected group bounds
        "lower": float,       # Lower bound
        "lower_inclusive": bool,
        "upper": float,       # Optional upper bound
        "upper_inclusive": bool
    },
    "reference": {            # Reference group bounds
        "lower": float,       # Optional lower bound
        "lower_inclusive": bool,
        "upper": float,       # Upper bound
        "upper_inclusive": bool
    }
}

2. For categorical features:

   {
       "feature": str,                  # Feature name
       "protected": str or int,         # Protected group category
       "reference": str or int          # Reference group category
   }
   

3. For probabilistic group membership:

   {
       "by_weights": True,
       "protected": str,         # Column name with protected group probabilities
       "reference": str          # Column name with reference group probabilities
   }default="test"
   

The dataset to evaluate fairness on.

metricstr, default=None

Fairness metric to calculate. Higher values indicate less unfairness. If None, defaults are used based on task type.

For regression (default=”SMD”):

• SMD: Standardized Mean Difference (%) between protected and reference groups

For classification (default=”AIR”):

• AIR: Adverse Impact Ratio of predicted probabilities

• PR: Precision Ratio

• RR: Recall Ratio

thresholdfloat or int, default=None

Optional threshold value to display in the visualization. Used to indicate acceptable fairness levels.

Returns:

ValidationResult

A container object with the following components:

• key: “compare_fairness”

• data: Name of the dataset used

• model: List of model names compared

• inputs: Input parameters used

• value: Dictionary of (“<model_name>”, item) pairs, which item is a nested dictionary with dictionary containing the (“<group_name>”, sub_item) pairs for each group; each sub_item contains

  • “fairness_metric”: the fairness metric for each group.

  • “distance”: The KS distance between protected vs reference group predictions.

  • “data_info”: A dictionary containing detailed information about the protected and reference groups, including sample indices and names.

    data_results = ds.data_drift_test(**results.value["MoLGBMClassifier"]["Gender"]["data_info"])
    data_results.plot("summary")
    data_results.plot(("density", "PAY_1"))
    

• table: DataFrame with detailed fairness metrics

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

  • “fairness”: a bar plot where x-axis is the group names, and y-axis is fairness metric

  • “distance”: a bar plot where x-axis is the group names, and y-axis is KS distance metric

Examples

compare_reliability(train_dataset: str = 'test', test_dataset: str = 'test', test_size: float = 0.5, alpha: float = 0.1, max_depth: int = 5, random_state: int = 0)

Compares reliability performance of multiple models under data shifts by evaluating prediction intervals/sets.

This function evaluates the prediction intervals/sets of various models by comparing their reliability metrics on specified training and testing datasets. It aggregates results such as average width and coverage for each model, providing insights into their performance under different conditions.

Parameters:

train_dataset{“main”, “train”, “test”}, default=”test”

Dataset used for model training and calibration. Choose from available dataset splits.

test_dataset{“main”, “train”, “test”}, default=”test”

Dataset used for evaluation. Choose from available dataset splits.

test_sizefloat, default=0.5

Proportion of data to use as test set when train_dataset equals test_dataset. Must be between 0 and 1.

alphafloat, default=0.1

Target miscoverage rate for prediction intervals/sets. Must be between 0 and 1.

max_depthint, default=5

Maximum depth of the GBM trees used for quantile regression. Only applicable for regression tasks.

random_stateint, default=0

Random seed for reproducible results.

Returns:

ValidationResult

A container object with the following components:

• key: “compare_reliability”

• data: Name of the dataset used

• model: List of model names being compared

• inputs: Input parameters used for the analysis

• value: Dictionary of (“<model_name>”, item), each item is also a dictionary with:

  • “interval”: Prediction intervals / sets and related metrics

  • “data_info”: The sample indices of reliable and unreliable samples, which can be further used for data distribution test, e.g.,

    data_results = ds.data_drift_test(**results.value["LGBMRegressor"]["data_info"])
    data_results.plot("summary")
    data_results.plot(("density", "MedInc"))
    

• table: DataFrame with detailed reliability metrics including average width and coverage for each model

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

  • width: Bar plot comparing empirical widths across models

  • coverage: Bar plot comparing empirical coverage across models

Examples

compare_residual_cluster(dataset: str = 'test', response_type: str = 'abs_residual', metric: str = None, n_clusters: int = 10, cluster_method: str = 'ltc', kmedoids_method: str = 'pam', sample_size: int = 2000, n_estimators: int = 100, max_depth: int = 5, random_state: int = 0, n_repeats: int = 10, perturb_features: str | Tuple = None, perturb_method: str = 'normal', noise_level: float | int = 0.1, alpha: float = 0.1)

Compare model residuals by clustering data points and evaluating performance within clusters.

This test identifies groups of samples with similar residual patterns by clustering data points based on their learning trajectories or proximity in feature space. It helps diagnose model performance heterogeneity across different data regions and identify problematic clusters where the model performs poorly.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

Dataset to analyze.

response_type: str, default=”abs_residual”

The response type, options include

• “abs_residual”: absolute residual

• “sq_residual”: squared residual

• “abs_residual_perturb”: absolute residual after X perturbation as used in robustness test

• “sq_residual_perturb”: squared residual after X perturbation

• “pi_width”: prediction interval width obtained as used in reliability test; note that as dataset=”test”, the test data will be split for calibration (conformal prediction), so that the calibration set is excluded in the final reported results.

metricstr, metric=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

n_clustersint, default=10

Number of clusters to create.

cluster_method{‘ltc’, ‘rf’}, default=’ltc’

Which algorithm to use.

• ‘ltc’: This method (Learning Trajectory Cluster; LTC) fits a gradient boosting models between predictors and the response_type. It extracts prediction trajectories during training, applies optional weighting schemes, performs PCA dimensionality reduction, and clusters samples based on their learning patterns.

• ‘rf’: This method (Random Forest; RF) fits a Random Forest between predictors and the response_type. Then, it generates a proximity matrix based on fitted trees, and clusters the distance matrix using KMedoids.

kmedoids_method{‘alternate’, ‘pam’}, default=’pam’

Which algorithm to use in KMedoids. ‘alternate’ is faster while ‘pam’ is more accurate. Only used when cluster_method=’rf’.

sample_sizeint, default=2000

sample size for speedup the calculation of the proximity matrix and clustering. Only used when cluster_method=’rf’.

n_estimatorsint, default=100

Number of trees in the Random Forest (cluster_method=’rf’) or gradient boosting models (cluster_method=’ltc’).

max_depthint, default=5

Maximum depth of trees in the Random Forest (cluster_method=’rf’) or gradient boosting models (cluster_method=’ltc’).

random_stateint, default=0

Random seed for reproducibility.

response_kwargsdict, default={}

Addition arguments for calculating the response.

n_repeatsint, default=10

Number of times to repeat the perturbation test for each noise level. Only used as response_type=”abs_residual_perturb” or “sq_residual_perturb”.

perturb_featuresstr or tuple, default=None

Features to perturb during testing. If None, all features are perturbed. Can be a single feature name or list of feature names. Only used as response_type=”abs_residual_perturb” or “sq_residual_perturb”.

perturb_method{“normal”, “quantile”}, default=”normal”

Method to perturb numerical features:

• “normal”: Add Gaussian noise scaled by feature standard deviation

• “quantile”: Perturb in quantile space with uniform noise

Only used as response_type=”abs_residual_perturb” or “sq_residual_perturb”.

noise_levelfloat, default=0.1

Magnitude of perturbation to apply. Can be a single value or tuple of values.

• For “normal” method: Standard deviation multiplier

• For “quantile” method: Maximum quantile shift

Only used as response_type=”abs_residual_perturb” or “sq_residual_perturb”.

alphafloat, default=0.1

Target miscoverage rate (1 - confidence level). For example, alpha=0.1 aims for 90% coverage. Only used as response_type=”pi_width”.

Returns:

ValidationResult

Contains:

• key: “compare_residual_cluster”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the test

• table: DataFrame with performance metrics for each cluster

• options: Dictionary of visualizations configuration. Run results.plot(name=xxx) to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

  • “cluster_performance”: Bar plot visualizing the performance scores against each cluster.

  • “feature_importance”: feature importance plot.

Notes

When response_type = “pi_width” and dataset=”test”, the test data will be split for calibration (conformal prediction), so that the calibration set is excluded in the final reported results.

Examples

compare_resilience(dataset: str = 'test', method: str = 'worst-sample', metric: str = None, alphas: tuple = None, n_clusters: int = 10, random_state: int = 0)

Compare model resilience performance under data shifts across multiple models.

This function compares the performance of different models under data shifts by evaluating their resilience scores. It allows users to specify the dataset partition, performance metric, and method for identifying problematic samples, and it returns a comprehensive result encapsulating the resilience scores and performance metrics.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

The dataset partition to analyze.

metricstr, default=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

method{“worst-sample”, “worst-cluster”, “outer-sample”, “hard-sample”}, default=”worst-sample”

Strategy for identifying challenging samples:

• “worst-sample”: Ranks samples by their prediction error

• “worst-cluster”: Groups samples into clusters and identifies problematic clusters

• “outer-sample”: Uses PCA to detect statistical outliers

• “hard-sample”: Trains a metamodel to identify inherently difficult samples

alphastuple of float, default=None

Fraction of worst ratios within (0, 1]. If None, it defaults to (0.1, 0.2, 0.3, …, 0.9, 1.0).

n_clustersint, default=10

Number of clusters when using method=”worst-cluster”.

random_stateint, default=0

Random seed for reproducibility.

Returns:

ValidationResult

A container object with the following components:

• key: “compare_resilience”

• data: Name of the dataset used

• model: List of model names being compared

• inputs: Input parameters used

• value: Dictionary of (“<model_name>”, item) pairs, which item is also a dictionary with scores for different fractions of worst samples (0.1 to 1.0) and corresponding sample indices.

  • “interval”: Prediction intervals / sets and related metrics

  • “data_info”: The sample indices of reliable and unreliable samples, which can be further used for data distribution test, e.g.,

    data_results = ds.data_drift_test(**results.value["MoLGBMRegressor"][0.2]["data_info"])
    data_results.plot("summary")
    data_results.plot(("density", "MedInc"))
    

• table: DataFrame with performance metrics across different data fractions

• options: Dictionary of visualizations configuration for a mulit-line plot where x-axis is the worst fractions (0 to 1), and y-axis is performance metric. Run results.plot() to show this plot.

Examples

compare_robustness(dataset: str = 'test', metric: str = None, n_repeats: int = 10, perturb_features: str | Tuple = None, perturb_method: str = 'normal', noise_levels: float | int | Tuple = 0.1, random_state: int = 0)

Performs robustness testing by comparing model performance under different perturbation levels.

This function performs robustness testing by comparing the performance of different models when subjected to various levels of perturbation. It allows for the specification of the dataset, performance metric, and perturbation characteristics, and returns a comprehensive result encapsulating the robustness scores and visualizations.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

Dataset partition to be used for testing.

metricstr, default=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

n_repeatsint, default=10

Number of times to repeat the perturbation test.

perturb_featuresstr or tuple, default=None

Specific features to perturb. If None, all features are perturbed.

perturb_method{“normal”, “quantile”}, default=”normal”

Method to generate perturbations:

• “normal”: Gaussian noise

• “quantile”: Quantile-based perturbation

noise_levelsfloat or tuple, default=0.1

Magnitude of perturbation to apply. Can be single value or multiple levels.

random_stateint, default=0

Seed for random number generation to ensure reproducibility.

Returns:

ValidationResult

A container object with the following components:

• key: “compare_robustness”

• data: Name of the dataset used

• model: List of model names being compared

• inputs: Input parameters used for the test

• value: Dictionary of (“<model_name>”, item) pairs, which item is also a dictionary with

  • noise_level (e.g., 0.1, 0.2): Performance metrics across noise levels;

  • data_info: Sample indices for worst and remaining cases (only for the first noise level)

    data_results = ds.data_drift_test(**results.value["MoLGBMRegressor"][0.2]["data_info"])
    data_results.plot("summary")
    data_results.plot(("density", "MedInc"))
    

• table: DataFrame containing detailed performance metrics for each noise level

• options: Dictionary of visualizations configuration for a bar plot visualizing the performance scores against each cluster. Run results.plot() to show this plot.

Examples

compare_slicing_accuracy(features: str, dataset: str = 'test', metric: str = None, method: str = 'uniform', bins: int | Dict = 10, n_estimators: int = 1000, threshold: float | int = None)

Compares model performance across different data slices based on a specified feature.

This function evaluates the performance of multiple models on different slices of the dataset, allowing for a detailed comparison based on a specified feature. It computes performance metrics for each model across the defined slices and returns a structured result containing the analysis.

Parameters:

featuresstr

Name of the feature to use for data slicing.

dataset{“main”, “train”, “test”}, default=”test”

The data set to be tested.

metricstr, default=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

method{“uniform”, “quantile”, “auto-xgb1”, “precompute”}, default=”uniform”

Method for binning numerical features:

• “uniform”: Equal-width binning

• “quantile”: Equal-frequency binning

• “auto-xgb1”: XGBoost-based automatic binning

• “precompute”: Use pre-specified bin edges

binsint or dict, default=10

Controls binning granularity:

• If int: Number of bins for numerical features. For “quantile”, this is the maximum number of bins. For “auto-xgb1”, this sets XGBoost’s max_bin parameter.

• If dict: Manual bin specifications for each feature, only used with method=”precompute”. Format: {feature_name: array_of_bin_edges}. Example: {“X0”: [0.1, 0.5, 0.9]} Note: Cannot specify bins for categorical features.

n_estimatorsint, default=1000

Number of estimators for XGBoost when method=”auto-xgb1”

thresholdfloat or int, default=None

Threshold for filtering fairness metric results. If not specified, it will not be used.

Returns:

ValidationResult

A container object with the following components:

• key: “compare_slicing_accuracy”

• data: Name of the dataset used

• model: List of model names being compared

• inputs: Input parameters used for the analysis

• value: Dictionary of (“<model_name>”, item) pairs, where each item is a list of dict, containing the information about the performance metrics for each segment,

  • “Feature”: feature name

  • “Segment”: segment value (categorical) or segment range (numerical)

  • “Size”: number of samples in this segment

  • <”metric”>: performance metric value of this segment

  • “Sample_ID”: sample indices of this segment

  • “Sample_Dataset”: dataset name, e.g., “train”, “test”, etc.

  • “Segment_Info”: explicit definition of this segment, similar to “Segment”

• table: Pandas DataFrame with detailed slicing statistics

• options: Dictionary of visualizations configuration for a mulit-line plot where x-axis is the selected slicing feature, and y-axis is performance metric. Run results.plot() to show this plot.

Examples

compare_slicing_fairness(group_config, features: str, favorable_label: int = 1, dataset: str = 'test', metric: str = None, method: str = 'uniform', bins: int | Dict = 10, n_estimators: int = 1000, threshold: float | int = None)

Evaluates fairness metrics across different protected and reference groups by slicing the data.

This function computes fairness metrics for specified groups in the dataset by analyzing the model’s predictions against the actual outcomes, allowing for a detailed comparison of fairness across different slices of the data.

Parameters:

group_configdict

Configuration defining protected and reference groups. Each key is a custom group name, and each value is a dictionary with group definitions. Supports three formats:

1. For numerical features:

   {

   “feature”: str, # Feature name “protected”: { # Protected group bounds

   “lower”: float, # Lower bound “lower_inclusive”: bool, “upper”: float, # Optional upper bound “upper_inclusive”: bool

   }, “reference”: { # Reference group bounds

   “lower”: float, # Optional lower bound “lower_inclusive”: bool, “upper”: float, # Upper bound “upper_inclusive”: bool

   }

   }

2. For categorical features:

   {

   “feature”: str, # Feature name “protected”: str or int, # Protected group category “reference”: str or int # Reference group category

   }

3. For probabilistic group membership:

   {

   “by_weights”: True, “protected”: str, # Column name with protected group probabilities “reference”: str # Column name with reference group probabilities

   }

featuresstr

Name of the feature to use for slicing the data

favorable_label{0, 1}, default=1

For classification: The preferred class label. For regression: 1 means larger predictions are preferred, 0 means smaller predictions are preferred.

dataset{“main”, “train”, “test”}, default=”test”

Which dataset partition to analyze

metricstr, default=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

method{“uniform”, “quantile”, “auto-xgb1”, “precompute”}, default=”uniform”

Method for binning numerical features

binsint or dict, default=10

Controls binning granularity:

• If int: Number of bins for numerical features. For “quantile”, this is the maximum number of bins. For “auto-xgb1”, this sets XGBoost’s max_bin parameter.

• If dict: Manual bin specifications for each feature, only used with method=”precompute”. Format: {feature_name: array_of_bin_edges}. Example: {“X0”: [0.1, 0.5, 0.9]} Note: Cannot specify bins for categorical features.

n_estimatorsint, default=1000

Number of estimators in xgboost, used when method=”auto-xgb1”.

thresholdfloat or int, default=None

Threshold for filtering fairness metric results. If not specified, it will not be used.

Returns:

ValidationResult

A container object with the following components:

• key: “compare_slicing_fairness”

• data: Name of the dataset used

• model: List of model names analyzed

• inputs: Input parameters used

• value: Dictionary of (“<model_name>”, item) pairs, and the item is also a dictionary with:

  • “<group_name>”: List of fairness metrics for each segment, and each element is a dict containing

    • “Feature”: feature name

    • “Segment”: segment value (categorical) or segment range (numerical)

    • “Size”: number of samples in this segment

    • <”metric”>: fairness metric value of this segment

    • “Sample_ID”: sample indices of this segment

    • “Sample_Dataset”: dataset name, e.g., “train”, “test”, etc.

    • “Segment_Info”: explicit definition of this segment, similar to “Segment”

    • “Weak”: boolean indicator showing whether this segment is weak or not

• table: dictionary of fairness metric table.

  • “<group_name>”: Table of fairness metrics for each segment.

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

  • “<group_name>”: Line plots visualizing the fairness metrics against the slicing feature.

Examples

compare_slicing_overfit(features: str, train_dataset: str = 'train', test_dataset: str = 'test', metric: str = None, method: str = 'uniform', bins: int | Dict = 10, n_estimators: int = 1000, threshold: float | int = None)

Compares model performance across different data slices to identify potential overfit regions.

This function evaluates the performance of multiple models on specified data slices, allowing for the identification of regions where a model may be overfitting. It utilizes various binning methods to segment the data based on the specified feature and computes performance metrics to assess the differences between training and testing datasets.

Parameters:

featuresstr

Name of the feature to use for data slicing.

train_dataset{“main”, “train”, “test”}, default=”train”

Specifies which dataset to use as the training set for comparison.

test_dataset{“main”, “train”, “test”}, default=”test”

Specifies which dataset to use as the test set for comparison.

metricstr, default=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

method{“uniform”, “quantile”, “auto-xgb1”, “precompute”}, default=”uniform”

Method for binning numerical features:

• “uniform”: Equal-width binning

• “quantile”: Equal-frequency binning

• “auto-xgb1”: XGBoost-based automatic binning

• “precompute”: Use pre-specified bin edges

binsint or dict, default=10

Controls binning granularity:

• If int: Number of bins for numerical features. For “quantile”, this is the maximum number of bins. For “auto-xgb1”, this sets XGBoost’s max_bin parameter.

• If dict: Manual bin specifications for each feature, only used with method=”precompute”. Format: {feature_name: array_of_bin_edges}. Example: {“X0”: [0.1, 0.5, 0.9]} Note: Cannot specify bins for categorical features.

n_estimatorsint, default=1000

Number of trees for XGBoost when using method=”auto-xgb1”

thresholdfloat or int, default=None

Threshold for filtering overfit regions. If not specified, it will not be used.

Returns:

ValidationResult

A container object with the following components:

• key: “compare_slicing_overfit”

• data: Name of the dataset used

• model: List of model names being compared

• inputs: Input parameters used for the analysis

• value: Dictionary of (“<model_name>”, item) pairs, where each item is a nested dictionary with dictionary containing the information about the performance metric gap for each segment,

  • “Feature”: feature name

  • “Segment”: segment value (categorical) or segment range (numerical)

  • “Size”: number of samples in this segment

  • <”metric”>: performance metric gap value of this segment

  • “Sample_ID”: sample indices of this segment

  • “Sample_Dataset”: dataset name, e.g., “train”, “test”, etc.

  • “Segment_Info”: explicit definition of this segment, similar to “Segment”

  • “Weak”: boolean indicator showing whether this segment is weak or not

• table: DataFrame with detailed performance metrics for each slice

• options: Dictionary of visualizations configuration for a mulit-line plot where x-axis is the selected slicing feature, and y-axis is performance metric gap. Run results.plot() to show this plot.

Examples

compare_slicing_reliability(features: str, train_dataset: str = 'test', test_dataset: str = 'test', test_size: float = 0.5, method: str = 'uniform', bins: int | Dict = 10, n_estimators: int = 1000, threshold: float | int = None, metric: str = 'width', alpha: float = 0.1, max_depth: int = 5, random_state: int = 0)

Compares reliability metrics across different slices of data for multiple models.

This function compares reliability metrics, such as width and coverage, across various slices of data for multiple models. It utilizes the specified features to segment the dataset and computes the reliability metrics based on the chosen method and parameters.

Parameters:

featuresstr

Name of the feature to use for slicing the data.

train_dataset{“main”, “train”, “test”}, default=”test”

Dataset to use for training and calibration purposes.

test_dataset{“main”, “train”, “test”}, default=”test”

Dataset to use for evaluation and comparison.

test_sizefloat, default=0.5

Proportion of data to use as test set when train_dataset equals test_dataset. Must be between 0 and 1.

method{“uniform”, “quantile”, “auto-xgb1”, “precompute”}, default=”uniform”

Method to use for creating bins:

• “uniform”: Equal-width bins

• “quantile”: Equal-frequency bins

• “auto-xgb1”: XGBoost-based automatic binning

• “precompute”: Use pre-defined bin edges

binsint or dict, default=10

Controls binning granularity:

• If int: Number of bins for numerical features. For “quantile”, this is the maximum number of bins. For “auto-xgb1”, this sets XGBoost’s max_bin parameter.

• If dict: Manual bin specifications for each feature, only used with method=”precompute”. Format: {feature_name: array_of_bin_edges}. Example: {“X0”: [0.1, 0.5, 0.9]} Note: Cannot specify bins for categorical features.

n_estimatorsint, default=1000

Number of trees for XGBoost when using method=”auto-xgb1”.

thresholdfloat or int, default=None

Threshold for filtering fairness unreliable regions. If not specified, it will not be used.

metric{“width”, “coverage”}, default=”width”

Reliability metric to compute:

• “width”: Average prediction interval width

• “coverage”: Average prediction coverage rate

alphafloat, default=0.1

Target coverage level for prediction intervals (between 0 and 1).

max_depthint, default=5

Maximum tree depth for gradient boosting model (regression tasks only).

random_stateint, default=0

Random seed for reproducibility.

Returns:

ValidationResult

A container object with the following components:

• key: “compare_slicing_reliability”

• data: Name of the dataset used

• model: List of model names being compared

• inputs: Input parameters used for the analysis

• value: Dictionary of (“<model_name>”, item) pairs, where each item is a nested dictionary with dictionary containing the information about the reliability metric for each segment.

  • “Feature”: feature name

  • “Segment”: segment value (categorical) or segment range (numerical)

  • “Size”: number of samples in this segment

  • <”metric”>: reliability metric value of this segment

  • “Sample_ID”: sample indices of this segment

  • “Sample_Dataset”: dataset name, e.g., “train”, “test”, etc.

  • “Segment_Info”: explicit definition of this segment, similar to “Segment”

  • “Weak”: boolean indicator showing whether this segment is weak or not

• table: DataFrame with detailed reliability statistics per slice

• options: Dictionary of visualizations configuration for a mulit-line plot where x-axis is the selected slicing feature, and y-axis is performance metric gap. Run results.plot() to show this plot.

Examples

compare_slicing_robustness(features: str, dataset: str = 'test', metric: str = None, method: str = 'uniform', bins: int | Dict = 10, n_estimators: int = 1000, threshold: float | int = None, n_repeats: int = 10, perturb_features: str | Tuple = None, perturb_method: str = 'normal', noise_levels: float | int = 0.1, random_state: int = 0)

Compares model robustness across different data slices by analyzing performance stability under perturbations.

This function evaluates the robustness of a model by analyzing its performance across various data slices, applying perturbations to the specified features, and measuring the stability of the model’s predictions. It allows for different binning methods and metrics to be used, providing flexibility in how the analysis is conducted.

Parameters:

featuresstr

Name of the feature to use for data slicing.

dataset{“main”, “train”, “test”}, default=”test”

Dataset to analyze.

method{“uniform”, “quantile”, “auto-xgb1”, “precompute”}, default=”uniform”

Method for binning numerical features:

• “uniform”: Equal-width bins

• “quantile”: Equal-frequency bins

• “auto-xgb1”: XGBoost-based automatic binning

• “precompute”: Use pre-defined bin edges

binsint or dict, default=10

Controls binning granularity:

• If int: Number of bins for numerical features. For “quantile”, this is the maximum number of bins. For “auto-xgb1”, this sets XGBoost’s max_bin parameter.

• If dict: Manual bin specifications for each feature, only used with method=”precompute”. Format: {feature_name: array_of_bin_edges}. Example: {“X0”: [0.1, 0.5, 0.9]} Note: Cannot specify bins for categorical features.

metricstr, default=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

n_estimatorsint, default=1000

The number of estimators in xgboost, used when method=”auto-xgb1”.

thresholdfloat or int, default=None

Threshold for filtering fairness non-robust regions. If not specified, it will not be used.

n_repeatsint, default=10

Number of times to repeat perturbation analysis.

perturb_featuresstr or tuple, default=None

Features to perturb. If None, all features are perturbed.

perturb_method{“normal”, “quantile”}, default=”normal”

Method for perturbing numerical features.

noise_levelsfloat or int, default=0.1

Magnitude of perturbation to apply.

random_stateint, default=0

Random seed for reproducibility.

Returns:

ValidationResult

A container object with the following components:

• key: “compare_slicing_robustness”

• data: Name of the dataset used

• model: List of model names being compared

• inputs: Input parameters used for the analysis

• value: Dictionary of (“<model_name>”, item) pairs, where each item is a nested dictionary with dictionary containing the information about the performance metric (after perturbation) for each segment.

  • “Feature”: feature name

  • “Segment”: segment value (categorical) or segment range (numerical)

  • “Size”: number of samples in this segment

  • <”metric”>: perturbed model performance metric value of this segment

  • “Sample_ID”: sample indices of this segment

  • “Sample_Dataset”: dataset name, e.g., “train”, “test”, etc.

  • “Segment_Info”: explicit definition of this segment, similar to “Segment”

  • “Weak”: boolean indicator showing whether this segment is weak or not

• table: DataFrame with comparative performance metrics

• options: Dictionary of visualizations configuration for a mulit-line plot where x-axis is the selected slicing feature, and y-axis is performance metric (after perturbation). Run results.plot() to show this plot.

Examples

delete_registed_test(name, run_id: str = None)

Load config and result of registered tests.

Parameters:

namestr

The name of test used for filtering.

run_idstr, default=None

run id of the registered test.

diagnose_accuracy_table(train_dataset: str = 'train', test_dataset: str = 'test', metric: str | Tuple = None)

Evaluate model performance on training and test datasets.

Calculates specified performance metrics for both training and test sets, along with the performance gap between them. This can help identify potential overfitting or underfitting issues.

For binary classification, the following plots will be generated.

• “roc_auc”: Generate the ROC AUC curve.

• “confusion_matrix”: Generate the confusion matrix.

• “precision_recall”: Generate the precision-recall curve.

Parameters:

train_dataset{“main”, “train”, “test”}, default=”train”

Dataset to use for training metrics. Must be one of the predefined dataset splits.

test_dataset{“main”, “train”, “test”}, default=”test”

Dataset to use for testing metrics. Must be one of the predefined dataset splits.

metricstr or tuple of str, default=None

Performance metric(s) to calculate. If None:

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

Returns:

ValidationResult

A result object containing:

• key: “diagnose_accuracy_table”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the test

• value: Dictionary of (“<metric_name>”, item), each item is also a dictionary with:

  • “<train_dataset>”: The metric value of training dataset.

  • “<test_dataset>”: The metric value of testing dataset.

  • “GAP”: The performance gap is calculated as (test_score - train_score).

• table: DataFrame of metric results

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

  • (“roc_auc”, “<train_dataset>”): roc curve for training dataset

  • (“roc_auc”, “<test_dataset>”): roc curve for testing dataset

  • (“precision_recall”, “<train_dataset>”): precision recall curve for training dataset

  • (“precision_recall”, “<test_dataset>”): precision recall for testing dataset

  • (“confusion_matrix”, “<train_dataset>”): roc curve for training dataset

  • (“confusion_matrix”, “<test_dataset>”): confusion matrix for testing dataset

Examples

diagnose_fairness(group_config, favorable_label: int = 1, dataset: str = 'test', metric: str = None, threshold: float | int = None)

Evaluate fairness metrics across different protected and reference groups.

This method calculates various fairness metrics based on the provided group configurations and model predictions, allowing for an assessment of fairness across specified groups in the dataset. It returns a result object containing the computed metrics, visualizations, and other relevant information.

Parameters:

group_configdict

Configuration defining protected and reference groups. Each key is a custom group name, and each value is a dictionary with group definitions. Supports three formats:

1. For numerical features:

   {
       "feature": str,           # Feature name
       "protected": {            # Protected group bounds
           "lower": float,       # Lower bound
           "lower_inclusive": bool,
           "upper": float,       # Optional upper bound
           "upper_inclusive": bool
       },
       "reference": {            # Reference group bounds
           "lower": float,       # Optional lower bound
           "lower_inclusive": bool,
           "upper": float,       # Upper bound
           "upper_inclusive": bool
       }
   }
   

2. For categorical features:

   {
       "feature": str,                  # Feature name
       "protected": str or int,         # Protected group category
       "reference": str or int          # Reference group category
   }
   

3. For probabilistic group membership:

   {
       "by_weights": True,
       "protected": str,         # Column name with protected group probabilities
       "reference": str          # Column name with reference group probabilities
   }
   

favorable_label{0, 1}, default=1

• For classification: The preferred class label.

• For regression: 1 means larger predictions are preferred, 0 means smaller predictions are preferred.

dataset{“main”, “train”, “test”}, default=”test”

The dataset to evaluate fairness on.

metricstr, default=None

Fairness metric to calculate. Higher values indicate less unfairness. If None, defaults are used based on task type.

For regression (default=”SMD”):

• SMD: Standardized Mean Difference (%) between protected and reference groups

For classification (default=”AIR”):

• AIR: Adverse Impact Ratio of predicted probabilities

• PR: Precision Ratio

• RR: Recall Ratio

thresholdfloat or int, default=None

Optional threshold value to display in the visualization. Used to indicate acceptable fairness levels.

Returns:

ValidationResult

A result object containing:

• key: “diagnose_fairness”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the test

• value: Nested dictionary containing the (“<group_name>”, item) pairs for each group, and the item is also a dictionary with:

  • “fairness_metric”: the fairness metric for each group.

  • “distance”: The KS distance between protected vs reference group predictions.

  • “data_info”: The sample indices of the protected and reference groups, which can be further used for data distribution test, e.g.,

  data_results = ds.data_drift_test(**results.value["Gender"]["data_info"])
  data_results.plot("summary")
  data_results.plot(("density", "MedInc"))
  

• table: DataFrame with fairness metric scores for each group.

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

  • “fairness”: Fairness metric bar plots visualizing the scores for each group.

  • “<group_name>”: Empirical cumulative distribution function plots for the protected and reference group predictions, allowing for visual comparison of distributions.

Examples

diagnose_mitigate_unfair_binning(group_config, favorable_label: int = 1, dataset: str = 'test', metric: str = None, performance_metric: str = None, binning_features: str | Tuple = None, binning_method: str = 'uniform', bins: int | Dict = 10)

Mitigate model unfairness through feature value binning.

This method attempts to reduce model unfairness by binning feature values, which can help smooth out predictions across different groups.

1. For numerical features:

   {
       "feature": str,           # Feature name
       "protected": {            # Protected group bounds
           "lower": float,       # Lower bound
           "lower_inclusive": bool,
           "upper": float,       # Optional upper bound
           "upper_inclusive": bool
       },
       "reference": {            # Reference group bounds
           "lower": float,       # Optional lower bound
           "lower_inclusive": bool,
           "upper": float,       # Upper bound
           "upper_inclusive": bool
       }
   }
   

2. For categorical features:

   {
       "feature": str,                  # Feature name
       "protected": str or int,         # Protected group category
       "reference": str or int          # Reference group category
   }
   

3. For probabilistic group membership:

   {
       "by_weights": True,
       "protected": str,         # Column name with protected group probabilities
       "reference": str          # Column name with reference group probabilities
   }
   

favorable_label : {0, 1}, default=1

• For classification: The preferred class label.

• For regression: 1 means larger predictions are preferred, 0 means smaller predictions are preferred.

dataset{“main”, “train”, “test”}, default=”test”

Which dataset partition to analyze

metricstr, default=None

Fairness metric to calculate. Higher values indicate less unfairness. If None, defaults are used based on task type.

For regression (default=”SMD”):

• SMD: Standardized Mean Difference (%) between protected and reference groups

For classification (default=”AIR”):

• AIR: Adverse Impact Ratio of predicted probabilities

• PR: Precision Ratio

• RR: Recall Ratio

performance_metricstr, default=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

binning_featuresstr or tuple, default=None

Features to apply binning to. If None, bins all features.

binning_method{“uniform”, “quantile”, “auto-xgb1”, “precompute”}, default=”uniform”

Method for binning numerical features:

• “uniform”: Equal-width bins

• “quantile”: Equal-frequency bins

• “auto-xgb1”: XGBoost prebinning

• “precompute”: Use pre-specified bin edges from bins parameter

binsint or dict, default=10

For int: Number of bins for numerical features; For dict: Pre-computed bin edges for each feature when binning_method=”precompute”. Example: {“feature1”: [0.1, 0.5, 0.9], “feature2”: [0.3, 0.7]}

Returns:

ValidationResult

Contains:

• key: “diagnose_mitigate_unfair_binning”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the test

• value: Nested dictionary containing the (“<feature_name>”, item) pairs for each feature, and the item is also a dictionary with:

  • “Performance”: Predictive performance scores after binning each feature

  • “Fairness”: Fairness scores after binning each feature

• table: dictionary of dataframe with perforamnce and fairness metric scores after binning.

  • “Fairness”: Fairness scores table

  • “Performance”: Predictive performance table

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

  • “<group_name>”: Line and bar plots visualizing the performance and fairness scores against each binning features.

Examples

diagnose_mitigate_unfair_thresholding(group_config, favorable_label: int = 1, dataset: str = 'test', metric: str = None, performance_metric: str = None, proba_cutoff: int | Tuple = None)

Attempts to mitigate model unfairness by applying feature value binning.

This method evaluates how binning feature values affects both model fairness and performance. It works by replacing values within each bin with the bin’s mean value, which can help reduce unfair treatment of similar instances.

Parameters:

group_configdict

Configuration defining protected and reference groups. Supports three formats:

1. For numerical features:

   {
       "feature": str,           # Feature name
       "protected": {            # Protected group bounds
           "lower": float,       # Lower bound
           "lower_inclusive": bool,
           "upper": float,       # Optional upper bound
           "upper_inclusive": bool
       },
       "reference": {            # Reference group bounds
           "lower": float,       # Optional lower bound
           "lower_inclusive": bool,
           "upper": float,       # Upper bound
           "upper_inclusive": bool
       }
   }
   

2. For categorical features:

   {
       "feature": str,                  # Feature name
       "protected": str or int,         # Protected group category
       "reference": str or int          # Reference group category
   }
   

3. For probabilistic group membership:

   {
       "by_weights": True,
       "protected": str,         # Column name with protected group probabilities
       "reference": str          # Column name with reference group probabilities
   }
   

favorable_label{0, 1}, default=1

• For classification: The preferred class label.

• For regression: 1 means larger predictions are preferred, 0 means smaller predictions are preferred.

dataset{“main”, “train”, “test”}, default=”test”

Which dataset partition to analyze

metricstr, optional

Fairness metric to use. Higher values indicate better fairness.

For regression (default=”SMD”):

• “SMD”: Standardized Mean Difference (%) between groups

For classification (default=”AIR”):

• “AIR”: Adverse Impact Ratio

• “PR”: Precision Ratio

• “RR”: Recall Ratio

performance_metricstr, default=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

proba_cutoff: int or tuple, default=20

If int, it represents the number of uniform grid points of cutoff values between 0 and 1. If tuple of float, it is the custom grid points of cutoff values, and its values should be within 0 and 1.

Returns:

ValidationResult

Contains:

• key: “diagnose_mitigate_unfair_thresholding”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the test

• value: Nested dictionary containing the (“<threshold>”, item) pairs for each threshold, and the item is also a dictionary with:

  • “Performance”: Predictive performance scores after adjusting the threshold

  • “Fairness”: Fairness scores after adjusting the threshold

• table: dictionary of dataframe with perforamnce and fairness metric scores after adjusting the threshold.

  • “Fairness”: Fairness scores table

  • “Performance”: Predictive performance table

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

  • “<group_name>”: Line plots visualizing the performance and fairness scores against each threshold.

Notes

The method compares baseline metrics (no binning) against metrics after binning each feature. This can help identify which features, when binned, most effectively balance fairness and performance tradeoffs.

Examples

diagnose_reliability(train_dataset: str = 'test', test_dataset: str = 'test', test_size: float = 0.5, alpha: float = 0.1, max_depth: int = 5, width_threshold: float = 0.1, random_state: int = 0)

Evaluates model reliability using split conformal prediction.

This method assesses the reliability of model predictions by generating prediction intervals (regression) or prediction sets (classification) using conformal prediction.

Parameters:

train_dataset{“main”, “train”, “test”}, default=”test”

Dataset used for calibrating the conformal prediction model. Not to be confused with the model’s original training set.

test_dataset{“main”, “train”, “test”}, default=”test”

Dataset used for evaluation.

test_sizefloat, default=0.5

Proportion of data to use for testing when train_dataset == test_dataset. Must be between 0 and 1.

alphafloat, default=0.1

Target miscoverage rate (1 - confidence level). For example, alpha=0.1 aims for 90% coverage.

max_depthint, default=5

Maximum depth of the gradient boosting trees for regression tasks. Only used when task_type is REGRESSION.

width_thresholdfloat, default=0.1

For regression: proportion of samples with largest prediction intervals to classify as unreliable.

random_stateint, default=0

Random seed for reproducibility.

Returns:

ValidationResult

A result object containing:

• key: “diagnose_reliability”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the test

• table: DataFrame with average width and coverage metrics

• value: Dictionary containing detailed results including:

  • “interval”: Prediction intervals / sets and related metrics

  • “data_info”: The sample indices of reliable and unreliable samples, which can be further used for data distribution test, e.g.,

    data_results = ds.data_drift_test(**results.value["data_info"])
    data_results.plot("summary")
    data_results.plot(("density", "MedInc"))
    

• options: Dictionary of visualizations configuration for a line plot where x-axis is the actual response, and y-axis is prediction, and prediction interval. Run results.plot() to show this plot.

Notes

For regression tasks:

• Uses residual quantile regression to calculate prediction intervals

• The calibration dataset is split 50/50 into training (for fitting quantile regression model) and validation (for calculating the threshold of nonconformity scores) sets

• Samples with widest prediction intervals are marked as unreliable

For classification tasks:

• Generates prediction sets: {0}, {1}, {0,1}, or {}

• Uses nonconformity scores to determine set membership

• Samples with prediction sets {} or {0,1} are marked as unreliable

Examples

diagnose_residual_analysis(features: str = None, use_prediction: bool = False, dataset: str = 'test', sample_size: int = 2000, random_state: int = 0)

Analyze the relationship between model residuals and a specified feature.

Creates a scatter plot showing the residuals (actual - predicted values) against a chosen feature or target variable. This can help identify patterns or heteroscedasticity in the model’s predictions. For classification tasks, residuals are calculated using predicted probabilities of the positive class.

Parameters:

featuresstr, default=None

The name of the feature to plot on the x-axis. Can be ignored when use_prediction is True.

use_prediction: bool, default=False

Whether to use the model prediction (predicted probability for classification) as x-axis.

dataset{“main”, “train”, “test”}, default=”test”

Which dataset partitionto use for the analysis.

sample_sizeint, default=2000

Maximum number of points to plot. If the dataset is larger, a random subsample of this size will be used to improve visualization clarity.

random_stateint, default=0

Random seed for reproducible subsampling.

Returns:

ValidationResult

A container object with the following attributes:

• key: “diagnose_residual”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the test

• value : Dictionary containing the x-axis values and residuals used in the plot

• table : DataFrame containing the plotted data

• options: Dictionary of visualizations configuration for a scatter plot where x-axis is the selected feature value, and y-axis is prediction residual (y - y_hat). Run results.plot() to show this plot.

Notes

For classification models, residuals are calculated as the difference between the actual class labels and the predicted probabilities for the positive class. For regression models, residuals are the difference between actual and predicted values.

Examples

diagnose_residual_cluster(dataset: str = 'test', response_type: str = 'abs_residual', metric: str = None, n_clusters: int = 10, cluster_method: str = 'ltc', kmedoids_method: str = 'pam', sample_size: int = 2000, n_estimators: int = 100, max_depth: int = 5, random_state: int = 0, n_repeats: int = 10, perturb_features: str | Tuple = None, perturb_method: str = 'normal', noise_level: float | int = 0.1, alpha: float = 0.1)

Analyze model residuals by clustering data points and evaluating performance within clusters.

This test identifies groups of samples with similar residual patterns by clustering data points based on their learning trajectories or proximity in feature space. It helps diagnose model performance heterogeneity across different data regions and identify problematic clusters where the model performs poorly.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

Dataset to analyze.

response_type: str, default=”abs_residual”

The response type, options include

• “abs_residual”: absolute residual

• “sq_residual”: squared residual

• “abs_residual_perturb”: absolute residual after X perturbation as used in robustness test

• “sq_residual_perturb”: squared residual after X perturbation

• “pi_width”: prediction interval width obtained as used in reliability test; note that as dataset=”test”, the test data will be split for calibration (conformal prediction), so that the calibration set is excluded in the final reported results.

metricstr, metric=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

n_clustersint, default=10

Number of clusters to create.

cluster_method{‘ltc’, ‘rf’}, default=’ltc’

Which algorithm to use.

• ‘ltc’: This method (Learning Trajectory Cluster; LTC) fits a gradient boosting models between predictors and the response_type. It extracts prediction trajectories during training, applies optional weighting schemes, performs PCA dimensionality reduction, and clusters samples based on their learning patterns.

• ‘rf’: This method (Random Forest; RF) fits a Random Forest between predictors and the response_type. Then, it generates a proximity matrix based on fitted trees, and clusters the distance matrix using KMedoids.

kmedoids_method{‘alternate’, ‘pam’}, default=’pam’

Which algorithm to use in KMedoids. ‘alternate’ is faster while ‘pam’ is more accurate. Only used when cluster_method=’rf’.

sample_sizeint, default=2000

sample size for speedup the calculation of the proximity matrix and clustering. Only used when cluster_method=’rf’.

n_estimatorsint, default=100

Number of trees in the Random Forest (cluster_method=’rf’) or gradient boosting models (cluster_method=’ltc’).

max_depthint, default=5

Maximum depth of trees in the Random Forest (cluster_method=’rf’) or gradient boosting models (cluster_method=’ltc’).

random_stateint, default=0

Random seed for reproducibility.

response_kwargsdict, default={}

Addition arguments for calculating the response.

n_repeatsint, default=10

Number of times to repeat the perturbation test for each noise level. Only used as response_type=”abs_residual_perturb” or “sq_residual_perturb”.

perturb_featuresstr or tuple, default=None

Features to perturb during testing. If None, all features are perturbed. Can be a single feature name or list of feature names. Only used as response_type=”abs_residual_perturb” or “sq_residual_perturb”.

perturb_method{“normal”, “quantile”}, default=”normal”

Method to perturb numerical features:

• “normal”: Add Gaussian noise scaled by feature standard deviation

• “quantile”: Perturb in quantile space with uniform noise

Only used as response_type=”abs_residual_perturb” or “sq_residual_perturb”.

noise_levelfloat, default=0.1

Magnitude of perturbation to apply. Can be a single value or tuple of values.

• For “normal” method: Standard deviation multiplier

• For “quantile” method: Maximum quantile shift

Only used as response_type=”abs_residual_perturb” or “sq_residual_perturb”.

alphafloat, default=0.1

Target miscoverage rate (1 - confidence level). For example, alpha=0.1 aims for 90% coverage. Only used as response_type=”pi_width”.

Returns:

ValidationResult

Contains:

• key: “diagnose_resilience_cluster”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the test

• value: Dict with clustering results including feature importance, embeddings, and per-cluster performance

  • “feature_importance”: residual feature importance by random forest

  • “cluster_X”: X being clustered

  • “cluster_y”: y being clustered

  • “cluster_sample_weight”: sample_weight being clustered

  • “cluster_labels”: cluster labels of each sample

  • “cluster_pred_func”: the function that receives X as input and output the cluster ID

  • “clusters”: Nested dict containing the detailed information about each cluster, the i-th cluster can be accessed via its cluster id, i.e., results.value[“cluster”][i], which includes items:

    • “score”: The performance metric of this cluster;

    • “data_info”: The sample indices within and outside this cluster, which can be further used for data distribution test, e.g.,

    data_results = ds.data_drift_test(**results.value["clusters"][2]["data_info"])
    data_results.plot("summary")
    data_results.plot(("density", "MedInc"))
    

• table: DataFrame with performance metrics for each cluster

• options: Dictionary of visualizations configuration. Run results.plot(name=xxx) to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

  • “cluster_residual”: Bar plot of residual for each cluster.

  • “cluster_performance”: Bar plot visualizing the performance scores against each cluster.

  • “feature_importance”: feature importance plot.

Notes

When response_type = “pi_width” and dataset=”test”, the test data will be split for calibration (conformal prediction), so that the calibration set is excluded in the final reported results.

Examples

diagnose_residual_interpret(dataset: str = 'test', n_estimators: int = 100, max_depth: int = 2, **xgb_kwargs)

Analyzes feature importance by examining their relationship with prediction residuals.

This method calculates how much each feature contributes to explaining the model’s prediction errors (residuals). A higher importance score indicates the feature has a stronger relationship with prediction errors.

As method is one of {“uniform”, “quantile”, “precompute”}, this test performs binning to each predictor variable, and then transform the binning results using one-hot encoding. The encoded varialbes are then fitted to the residual using l2-regularized linear model. The importance of each predictor (under the framework of functional ANOVA) are aggregated using the linear coefficients.

As the method is “auto-xgb1”, then a xgboost depth-1 model is used to fit predictors and the residual. And the feature importance of the xgboost model (under the framework of functional ANOVA) is used as final feature importance.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

Which dataset to analyze.

n_estimatorsint, default=100

Number of trees in Xgboost.

max_depthint, default=2

The maximum tree depth in Xgboost.

**xgb_kwargs:

Other hyperparameters for xgboost.

Returns:

ValidationResult

Contains:

• key: “diagnose_residual_interpret”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the test

• value: Dictionary containing:

  “Model”: the fitted xgboost model object “DataSet”: the dataset that contains residual as response “Feature Importance”: A dict with list of feature names and residual feature importance “Effect Importance”: A dict with list of effect names and residual effect importance

• options: Dictionary of visualizations configuration. Run results.plot(name=xxx) to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

  • “feature_importance”: feature importance plot.

  • “effect_importance”: effect importance plot.

Notes

The feature importance is calculated as the normalized variance of predictions when using each feature alone. For methods other than “auto-xgb1”, features are first binned then one-hot encoded before fitting a Ridge regression model to predict absolute residuals.

Examples

diagnose_resilience(dataset: str = 'test', method: str = 'worst-sample', metric: str = None, alphas: tuple = None, n_clusters: int = 10, random_state: int = 0)

Evaluate model resilience by analyzing performance on challenging data subsets.

This method assesses how well the model maintains its performance when faced with increasingly difficult or anomalous samples. It helps identify potential vulnerabilities and understand the model’s behavior under stress conditions.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

The dataset to use for resilience testing.

method{“worst-sample”, “worst-cluster”, “outer-sample”, “hard-sample”}, default=”worst-sample”

Strategy for identifying challenging samples:

• “worst-sample”: Ranks samples by their prediction error

• “worst-cluster”: Groups samples into clusters and identifies problematic clusters

• “outer-sample”: Uses PCA to detect statistical outliers

• “hard-sample”: Trains a metamodel to identify inherently difficult samples

metricstr, metric=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

alphastuple of float, default=None

Fraction of worst ratios within (0, 1]. If None, it defaults to (0.1, 0.2, 0.3, …, 0.9, 1.0).

n_clustersint, default=10

Number of clusters for the “worst-cluster” method. Ignored for other methods.

random_stateint, default=0

Random seed for reproducibility.

Returns:

ValidationResult

A result object containing:

• key: “diagnose_resilience”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the test

• value: Nested dict containing the detailed information about each worst ratio (alpha), e.g., results.value[0.2], which includes items:

  • “score”: The performance metric of the selected “worst samples”;

  • “data_info”: The sample indices within and outside this cluster, which can be further used for data distribution test, e.g.,

    data_results = ds.data_drift_test(**results.value[0.2]["data_info"])
    data_results.plot("summary")
    data_results.plot(("density", "MedInc"))
    

• table: DataFrame showing performance scores across different fractions

• options: Dictionary of visualizations configuration for a line plot where x-axis is the worst fractions (0 to 1), and y-axis is performance metric. Run results.plot() to show this plot.

Notes

The method evaluates model performance on increasingly larger subsets of the most challenging samples (from 10% to 100%). A steep performance drop indicates potential resilience issues in specific scenarios.

Examples

diagnose_robustness(dataset: str = 'test', threshold: float = 0.1, metric: str = None, n_repeats: int = 10, perturb_features: str | Tuple = None, perturb_method: str = 'normal', noise_levels: float | int | Tuple = 0.1, random_state: int = 0)

Evaluate model robustness by measuring performance under feature perturbations.

This test assesses how model predictions change when input features are perturbed with different noise levels. It helps identify the model’s stability and sensitivity to input variations.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

Dataset to evaluate robustness on.

metricstr, metric=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

n_repeatsint, default=10

Number of times to repeat the perturbation test for each noise level.

perturb_featuresstr or tuple, default=None

Features to perturb during testing. If None, all features are perturbed. Can be a single feature name or list of feature names.

perturb_method{“normal”, “quantile”}, default=”normal”

Method to perturb numerical features:

• “normal”: Add Gaussian noise scaled by feature standard deviation

• “quantile”: Perturb in quantile space with uniform noise

noise_levelsfloat or tuple, default=0.1

Magnitude of perturbation to apply. Can be a single value or tuple of values.

• For “normal” method: Standard deviation multiplier

• For “quantile” method: Maximum quantile shift

thresholdfloat, default=0.1

Proportion of samples to consider as “Non-robust” cases based on prediction changes. Used for separating samples into “Non-robust” and “Remaining” groups.

random_stateint, default=0

Random seed for reproducible results.

Returns:

ValidationResult

Object containing:

• key: “diagnose_robustness”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the test

• value: Nested dict containing the detailed information about each noise level, e.g., results.value[0.2], which includes items:

  • “score”: The performance metric after perturbing the data with noise level 0.2;

  • “data_info”: The sample indices of small and large prediction changes groups (determined by the threshold), which can be further used for data distribution test, e.g.,

    data_results = ds.data_drift_test(**results.value[0.2]["data_info"])
    data_results.plot("summary")
    data_results.plot(("density", "MedInc"))
    

• table: pd.DataFrame of performance under different noise level and repeat

• options: Dictionary of visualizations configuration for a box plot where x-axis is the noise level, and y-axis is performance metric. Run results.plot() to show this plot.

Examples

diagnose_slicing_accuracy(features: str | Tuple = None, dataset: str = 'test', metric: str = None, method: str = 'uniform', bins: int | Dict = 10, n_estimators: int = 1000, threshold: float | int = None)

Identify low-accuracy regions based on specified slicing features.

This method analyzes the performance of a model on specified features and identifies regions where the model exhibits low accuracy. It supports both 1D and 2D slicing based on the input features.

Parameters:

featuresUnion[str, Tuple], default=None

Feature names used for slicing. Each tuple element should contain at most 2 features.

• If features=(“X1”, ) or “X1”, computes 1D slicing over X1.

• If features=(“X1”, “X2”), computes 2D slicing over the interaction of X1 and X2.

• If features=((“X1”, ), (“X2”, )), computes 1D slicing over X1 and X2 separately.

Note: Batch mode for 2D slicing is not supported. If None, all 1D features will be used.

datasetstr, default=”test”

The dataset to be tested. Options are “main”, “train”, or “test”.

metricstr, metric=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

method{“uniform”, “quantile”, “auto-xgb1”, “precompute”}, default=”uniform”

Method for binning numerical features:

• “uniform”: Equal-width binning

• “quantile”: Equal-frequency binning (may result in fewer bins due to ties)

• “auto-xgb1”: Use bins of a XGBoost depth-1 model fitted between X and residuals.

• “precompute”: Uses pre-specified bin edges

Note that for uniform, quantile, and precompute, all variables including inactive ones can be used for spliting. But for auto-xgb1, only use active features (X) for fitting XGB.

binsint or dict, default=10

Controls binning granularity:

• If int: Number of bins for numerical features. For “quantile”, this is the maximum number of bins. For “auto-xgb1”, this sets XGBoost’s max_bin parameter.

• If dict: Manual bin specifications for each feature, only used with method=”precompute”. Format: {feature_name: array_of_bin_edges}. Example: {“X0”: [0.1, 0.5, 0.9]} Note: Cannot specify bins for categorical features.

n_estimatorsint, default=1000

The number of estimators in XGBoost, applicable when method=”auto-xgb1”.

thresholdfloat or int, default=None

The metric threshold for identifying weak regions. If not specified, it will be the metric of the whole population.

Returns:

ValidationResult

The result of the Slicing Accuracy detection, including key metrics and tables.

• key: “diagnose_slicing_accuracy”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the test

• value: List of performance metrics for each segment, and each element is a dict containing

  • “Feature”: feature name

  • “Segment”: segment value (categorical) or segment range (numerical)

  • “Size”: number of samples in this segment

  • <”metric”>: performance metric value of this segment

  • “Sample_ID”: sample indices of this segment

  • “Sample_Dataset”: dataset name, e.g., “train”, “test”, etc.

  • “Segment_Info”: explicit definition of this segment, similar to “Segment”

  • “Weak”: boolean indicator showing whether this segment is weak or not

• table: pd.DataFrame summarizing the results, including features, segments, sizes, and the specified metric.

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; To display one preferred plot by results.plot(name=xxx), and the following names are available:

  • None (If only one 1D or 2D slicing features are specified): Performance metric plot against selected slicing feature(s).

  • “<feature_name>” (If multiple single features are specified): Performance metric plot against selected slicing feature(s).

Examples

diagnose_slicing_fairness(group_config, features: str | Tuple = None, favorable_label: int = 1, dataset: str = 'test', metric: str = None, method: str = 'uniform', bins: int | Dict = 10, n_estimators: int = 1000, threshold: float | int = None)

Evaluate a model’s slicing fairness metric across different protected-reference groups.

This function assesses the fairness of a model by calculating specified metrics across various protected and reference groups defined in the group_config. It takes into account the features used for slicing, the dataset to be evaluated, and the method for binning numerical features, among other parameters. The results include a validation object containing the fairness metrics and related information.

Parameters:

group_configdict

Configuration defining protected and reference groups. Each key is a custom group name, and each value is a dictionary with group definitions. Supports three formats:

1. For numerical features:

   {
       "feature": str,           # Feature name
       "protected": {            # Protected group bounds
           "lower": float,       # Lower bound
           "lower_inclusive": bool,
           "upper": float,       # Optional upper bound
           "upper_inclusive": bool
       },
       "reference": {            # Reference group bounds
           "lower": float,       # Optional lower bound
           "lower_inclusive": bool,
           "upper": float,       # Upper bound
           "upper_inclusive": bool
       }
   }
   

2. For categorical features:

   {
       "feature": str,                  # Feature name
       "protected": str or int,         # Protected group category
       "reference": str or int          # Reference group category
   }
   

3. For probabilistic group membership:

   {
       "by_weights": True,
       "protected": str,         # Column name with protected group probabilities
       "reference": str          # Column name with reference group probabilities
   }
   

featuresUnion[str, Tuple], default=None

Feature names used for slicing. Each tuple element should contain at most 2 features.

• If features=(“X1”, ) or “X1”, computes 1D slicing over X1.

• If features=(“X1”, “X2”), computes 2D slicing over the interaction of X1 and X2.

• If features=((“X1”, ), (“X2”, )), computes 1D slicing over X1 and X2 separately.

Note: Batch mode for 2D slicing is not supported. If None, all 1D features will be used.

favorable_label{0, 1}, default=1

• For classification: The preferred class label.

• For regression: 1 means larger predictions are preferred, 0 means smaller predictions are preferred.

dataset{“main”, “train”, “test”}, default=”train”

The dataset to be tested.

metric{“AIR”, “SMD”, “PR”, “RR”}, default=None

The fairness metric(s) to calculate. If None, defaults to SMD for regression and AIR for classification.

method{“uniform”, “quantile”, “auto-xgb1”, “precompute”}, default=”uniform”

Method for binning numerical features:

• “uniform”: Equal-width binning

• “quantile”: Equal-frequency binning (may result in fewer bins due to ties)

• “auto-xgb1”: Use bins of a XGBoost depth-1 model fitted between X and residuals.

• “precompute”: Uses pre-specified bin edges

binsint or dict, default=10

Controls binning granularity:

• If int: Number of bins for numerical features. For “quantile”, this is the maximum number of bins. For “auto-xgb1”, this sets XGBoost’s max_bin parameter.

• If dict: Manual bin specifications for each feature, only used with method=”precompute”. Format: {feature_name: array_of_bin_edges}. Example: {“X0”: [0.1, 0.5, 0.9]} Note: Cannot specify bins for categorical features.

n_estimatorsint, default=1000

Number of estimators in xgboost, used when method=”auto-xgb1”.

thresholdfloat or int, default=None

Threshold for filtering fairness metric results. If not specified, it will be the fairness metric of the whole population of each group, respectively.

Returns:

ValidationResult

Slicing Fairness result, which includes:

• key: “diagnose_slicing_fairness”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the test

• value: Nested dictionary containing the (“<feature_name>”, item) pairs for each feature (this level is only used in batch mode, i.e., multiple 1D features are specified), and the item is also a dictionary with:

  • “<group_name>”: List of fairness metrics for each segment, and each element is a dict containing

    • “Feature”: feature name

    • “Segment”: segment value (categorical) or segment range (numerical)

    • “Size”: number of samples in this segment

    • <”metric”>: fairness metric value of this segment

    • “Sample_ID”: sample indices of this segment

    • “Sample_Dataset”: dataset name, e.g., “train”, “test”, etc.

    • “Segment_Info”: explicit definition of this segment, similar to “Segment”

    • “Weak”: boolean indicator showing whether this segment is weak or not

• table: dictionary of fairness metric table.

  • “<group_name>”: Table of fairness metrics for each segment.

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; To display one preferred plot by results.plot(name=xxx), and the following names are available:

  • “<group_name>” (If only one 1D or 2D slicing features are specified): Fairness metric plots against selected slicing feature(s).

  • “(<feature_name>, <group_name>)” (If multiple single features are specified): Fairness metric plots against selected slicing feature(s).

Examples

diagnose_slicing_overfit(features: str | Tuple = None, train_dataset: str = 'train', test_dataset: str = 'test', metric: str = None, method: str = 'uniform', bins: int | Dict = 10, n_estimators: int = 1000, threshold: float | int = None)

Identify overfit regions based on one or two slicing features.

This method analyzes the performance gap between training and testing datasets for specified features, helping to identify potential overfitting.

Parameters:

featuresUnion[str, Tuple], default=None

Feature names used for slicing. Each tuple element should contain at most 2 features.

• If features=(“X1”, ) or “X1”, computes 1D slicing over X1.

• If features=(“X1”, “X2”), computes 2D slicing over the interaction of X1 and X2.

• If features=((“X1”, ), (“X2”, )), computes 1D slicing over X1 and X2 separately.

Note: Batch mode for 2D slicing is not supported. If None, all 1D features will be used.

train_datasetstr, default=”train”

The dataset used for training. Options include “main”, “train”, or “test”.

test_datasetstr, default=”test”

The dataset used for testing. Options include “main”, “train”, or “test”.

metricstr, metric=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

methodstr, default=”uniform”

The binning method for numerical features. Options include:

• “uniform”: Equal-width bins

• “quantile”: Equal-frequency bins

• “auto-xgb1”: Binning method for XGBoost

• “precompute”: Predefined bins

binsint or dict, default=10

Controls binning granularity:

• If int: Number of bins for numerical features. For “quantile”, this is the maximum number of bins. For “auto-xgb1”, this sets XGBoost’s max_bin parameter.

• If dict: Manual bin specifications for each feature, only used with method=”precompute”. Format: {feature_name: array_of_bin_edges}. Example: {“X0”: [0.1, 0.5, 0.9]} Note: Cannot specify bins for categorical features.

n_estimatorsint, default=1000

The number of estimators to use in XGBoost when method=”auto-xgb1”.

thresholdfloat or int, default=None

The metric gap threshold for identifying weak regions. If not specified, it will be the performance metric gap of the whole population.

Returns:

ValidationResult

An object containing the results of the slicing overfit detection, including:

• key: “diagnose_slicing_overfit”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the test

• value: List of performance metrics for each segment, and each element is a dict containing

  • “Feature”: feature name

  • “Segment”: segment value (categorical) or segment range (numerical)

  • “Size”: number of samples in this segment

  • <”metric”>: performance metric gap value of this segment

  • “Sample_ID”: sample indices of this segment

  • “Sample_Dataset”: dataset name, e.g., “train”, “test”, etc.

  • “Segment_Info”: explicit definition of this segment, similar to “Segment”

  • “Weak”: boolean indicator showing whether this segment is weak or not

• table: pd.DataFrame summarizing the performance metrics for both training and testing datasets, including the calculated gaps.

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; To display one preferred plot by results.plot(name=xxx), and the following names are available:

  • None (If only one 1D or 2D slicing features are specified): Performance gap plot against selected slicing feature(s).

  • “<feature_name>” (If multiple single features are specified): Performance gap plot against selected slicing feature(s).

Examples

diagnose_slicing_reliability(features: str | Tuple = None, train_dataset: str = 'test', test_dataset: str = 'test', test_size: float = 0.5, method: str = 'uniform', bins: int | Dict = 10, n_estimators: int = 1000, threshold: float | int = None, metric: str = 'width', alpha: float = 0.1, max_depth: int = 5, random_state: int = 0)

Get unreliable regions based on one or two slicing features.

This function analyzes the reliability of a model’s predictions by slicing the dataset based on the provided features. It computes metrics such as width or coverage for the specified bins and identifies regions where the model’s predictions may be unreliable.

Parameters:

featuresUnion[str, Tuple], default=None

Feature names used for slicing. Each tuple element should contain at most 2 features.

• If features=(“X1”, ) or “X1”, computes 1D slicing over X1.

• If features=(“X1”, “X2”), computes 2D slicing over the interaction of X1 and X2.

• If features=((“X1”, ), (“X2”, )), computes 1D slicing over X1 and X2 separately.

Note: Batch mode for 2D slicing is not supported. If None, all 1D features will be used.

train_dataset{“main”, “train”, “test”}, default=”test”

The data set used for training and calibration.

test_dataset{“main”, “train”, “test”}, default=”test”

The data set used for evaluation purpose.

test_sizefloat, default=0.5

Optional test set percentage for splitting the data into train and test sets. Only used when train_dataset == test_dataset.

method{“uniform”, “quantile”, “auto-xgb1”, “precompute”}, default=”uniform”

Method for binning numerical features:

• “uniform”: Equal-width binning

• “quantile”: Equal-frequency binning (may result in fewer bins due to ties)

• “auto-xgb1”: Use bins of a XGBoost depth-1 model fitted between X and residuals.

• “precompute”: Uses pre-specified bin edges

binsint or dict, default=10

Controls binning granularity:

• If int: Number of bins for numerical features. For “quantile”, this is the maximum number of bins. For “auto-xgb1”, this sets XGBoost’s max_bin parameter.

• If dict: Manual bin specifications for each feature, only used with method=”precompute”. Format: {feature_name: array_of_bin_edges}. Example: {“X0”: [0.1, 0.5, 0.9]} Note: Cannot specify bins for categorical features.

n_estimatorsint, default=1000

The number of estimators in xgboost, used when method=”auto-xgb1”.

thresholdfloat or int, default=None

The metric threshold of unreliable regions. If not specified, it will be the reliability metric of the whole population.

metric{“width”, “coverage”}, default=”width”

The metric to be calculated in each slicing bin.

• “width”: The average width in each bin.

• “coverage”: The average coverage in each bin.

alphafloat, default=0.1

The expected coverage of prediction intervals / sets.

max_depthint, default=5

The max_depth parameter of GBM with quantile loss. Only used for regression tasks.

random_stateint, default=0

The random seed for reproducibility.

Returns:

ValidationResult

A container object with the following components:

• key: “diagnose_slicing_reliability”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the analysis

• value: List of performance metrics for each segment, and each element is a dict containing

  • “Feature”: feature name

  • “Segment”: segment value (categorical) or segment range (numerical)

  • “Size”: number of samples in this segment

  • <”metric”>: reliability metric value of this segment

  • “Sample_ID”: sample indices of this segment

  • “Sample_Dataset”: dataset name, e.g., “train”, “test”, etc.

  • “Segment_Info”: explicit definition of this segment, similar to “Segment”

  • “Weak”: boolean indicator showing whether this segment is weak or not

• table: DataFrame with slice analysis results

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; To display one preferred plot by results.plot(name=xxx), and the following names are available:

  • None (If only one 1D or 2D slicing features are specified): Reliability metric plot against selected slicing feature(s).

  • “<feature_name>” (If multiple single features are specified): Reliability metric plot against selected slicing feature(s).

Examples

diagnose_slicing_robustness(features: str | Tuple = None, dataset: str = 'test', method: str = 'uniform', bins: int | Dict = 10, metric: str = None, n_estimators: int = 1000, threshold: float | int = None, n_repeats: int = 10, perturb_features: str | Tuple = None, perturb_method: str = 'normal', noise_levels: float | int = 0.1, random_state: int = 0)

Get unreliable regions based on one or two slicing features.

This function evaluates the robustness of a model by analyzing its performance across different slices of the dataset defined by the specified features. It computes the metric scores for the slices and identifies regions where the model’s performance is unreliable, allowing for insights into the model’s behavior under various conditions.

Parameters:

featuresUnion[str, Tuple], default=None

Feature names used for slicing. Each tuple element should contain at most 2 features.

• If features=(“X1”, ) or “X1”, computes 1D slicing over X1.

• If features=(“X1”, “X2”), computes 2D slicing over the interaction of X1 and X2.

• If features=((“X1”, ), (“X2”, )), computes 1D slicing over X1 and X2 separately.

Note: Batch mode for 2D slicing is not supported. If None, all 1D features will be used.

dataset{“main”, “train”, “test”}, default=”test”

The data set to be tested.

metricstr, metric=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

method{“uniform”, “quantile”, “auto-xgb1”, “precompute”}, default=”uniform”

Method for binning numerical features:

• “uniform”: Equal-width binning

• “quantile”: Equal-frequency binning (may result in fewer bins due to ties)

• “auto-xgb1”: Use bins of a XGBoost depth-1 model fitted between X and residuals.

• “precompute”: Uses pre-specified bin edges

binsint or dict, default=10

Controls binning granularity:

• If int: Number of bins for numerical features. For “quantile”, this is the maximum number of bins. For “auto-xgb1”, this sets XGBoost’s max_bin parameter.

• If dict: Manual bin specifications for each feature, only used with method=”precompute”. Format: {feature_name: array_of_bin_edges}. Example: {“X0”: [0.1, 0.5, 0.9]} Note: Cannot specify bins for categorical features.

n_estimatorsint, default=1000

The number of estimators in xgboost, used when method=”auto-xgb1”.

thresholdfloat or int, default=None

The metric threshold of non-robust regions. If not specified, it will be the robustness metric of the whole population.

n_repeatsint, default=10

The number of perturbation repetition.

perturb_featuresstr or tuple, default=None

Feature names used for perturbation. If None, all features will be perturbed.

perturb_method{“normal”, “quantile”}, default=”normal”

The perturbation method of numerical features.

noise_levelsfloat or int, default=0.1

The perturbation level.

random_stateint, default=0

The random seed for reproducibility.

Returns:

ValidationResult

A container object with the following components:

• key: “diagnose_slicing_robustness”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the analysis

• value: List of performance metrics for each segment, and each element is a dict containing

  • “Feature”: feature name

  • “Segment”: segment value (categorical) or segment range (numerical)

  • “Size”: number of samples in this segment

  • <”metric”>: perturbed model performance metric value of this segment

  • “Sample_ID”: sample indices of this segment

  • “Sample_Dataset”: dataset name, e.g., “train”, “test”, etc.

  • “Segment_Info”: explicit definition of this segment, similar to “Segment”

  • “Weak”: boolean indicator showing whether this segment is weak or not

• table: DataFrame summarizing slice-wise results

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; To display one preferred plot by results.plot(name=xxx), and the following names are available:

  • None (If only one 1D or 2D slicing features are specified): Performance metric (after perturbation) plot against selected slicing feature(s).

  • “<feature_name>” (If multiple single features are specified): Performance metric (after perturbation) plot against selected slicing feature(s).

Examples

diagnose_weakness_region(train_dataset='train', test_dataset='test', metric=None, geometry_method='arf', bins=10, weak_fraction=0.2, top_n_features=10, min_count=20, geometry_n_estimators=120, geometry_max_depth=12, geometry_min_samples_leaf=20, geometry_num_trees=30, geometry_max_iters=10, mi_n_estimators=200, mi_max_depth=0, mi_min_samples_leaf=10, mi_n_splits=5, random_state=0)

Diagnose model weakness regions using AMIF (Adversarial Mutual Information Forest).

AMIF partitions data into a 2D grid (geometry score x MI score) and identifies regions where the model performs poorly. For each grid cell, the model metric is computed on train and test subsets. Weak regions are those in the bottom fraction of test performance.

Features are ranked by Jensen-Shannon divergence between the weak-region distribution and the rest, highlighting which features differ most in the regions where the model struggles.

Parameters:

train_dataset{“main”, “train”, “test”}, default=”train”

Dataset split used for training performance.

test_dataset{“main”, “train”, “test”}, default=”test”

Dataset split used for test performance.

metricstr, default=None

Metric name. Defaults to “MSE” for regression, “ACC” for classification.

geometry_method{“arf”, “carf”}, default=”arf”

Geometry scoring method. “arf” uses Adversarial Random Forest (arfpy), “carf” uses Custom Adversarial Random Forest.

binsint, default=10

Number of quantile bins for each axis of the 2D grid.

weak_fractionfloat, default=0.2

Fraction of test bins (by metric) considered weak.

top_n_featuresint, default=10

Number of top features to show in JS divergence ranking.

min_countint, default=20

Minimum sample count per bin to compute metric.

geometry_n_estimatorsint, default=120

Number of trees for CARF geometry scorer.

geometry_max_depthint, default=12

Max depth for CARF geometry scorer.

geometry_min_samples_leafint, default=20

Min samples per leaf for CARF geometry scorer.

geometry_num_treesint, default=30

Number of trees for ARF geometry scorer.

geometry_max_itersint, default=10

Max adversarial iterations for ARF geometry scorer.

mi_n_estimatorsint, default=200

Number of trees for MI scorer.

mi_max_depthint, default=0

Max depth for MI scorer. 0 means unlimited (None).

mi_min_samples_leafint, default=10

Min samples per leaf for MI scorer.

mi_n_splitsint, default=5

Number of cross-validation splits for MI scorer.

random_stateint, default=0

Random seed for reproducibility.

Returns:

ValidationResult

Contains:

• table: DataFrame with columns [Geometry Bin, MI Bin, {metric} (Train), {metric} (Test), Sample Count, Is Weak], one row per populated bin.

• value: Dict with geometry_scores, mi_scores, bins, heatmaps, weak_bins, JS rankings, etc.

• options: Dict of mocharts visualizations including heatmaps of region performance, score distributions, JS divergence ranking, MI importance ranking, per-feature distributions, and confusion matrices (classification only).

display_test_results(testsuite_name, test_list: list = None)

Get ValidationResult object of registered test.

Parameters:

testsuite_namestr

The testsuite to display.

test_listlist

The content list of test, e.g.,

[{'name': 'test1', 'run_id': 'xxx', 'display_table': True, 'display_plot': True},
 {'name': 'test2', 'display_table': True, 'display_plot': False}]

or

['test1', 'test2']

explain_ale(features: str | Tuple[str] = None, dataset: str = 'test', sample_size: int = 5000, grid_resolution: int = 20, response_method: str = 'auto', random_state: int = 0)

Calculate Accumulated Local Effects (ALE) plots for one or two features.

ALE plots show how individual features influence model predictions while accounting for feature interactions by measuring effects locally rather than assuming independence across features as in partial dependence.

Parameters:

featuresstr or tuple of str

Feature name(s) to analyze. Use a single feature name for 1D ALE plot or a tuple of two feature names for 2D ALE plot. For 2D ALE, categorical features are not supported.

dataset{“main”, “train”, “test”}, default=”test”

Dataset to use for calculating the explanation results.

sample_sizeint, default=5000

Number of random samples to use for calculation. If None, uses entire dataset. Smaller samples speed up calculation but may reduce accuracy.

grid_resolutionint, default=20

Number of intervals to divide feature range for ALE calculation. Higher values give finer granularity but increase computation time.

response_method{“auto”, “decision_function”, “predict_proba”}, default=”auto”

Prediction method to use for binary classification tasks:

• “auto”: Uses ‘predict_proba’ if available, otherwise ‘decision_function’

• “predict_proba”: Probability of the positive class

• “decision_function”: Model’s decision function output

random_stateint, default=0

Random seed for reproducible sampling when sample_size is specified.

Returns:

ValidationResult

Object containing:

• key: “explain_ale”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the analysis

• value: Dictionary containing

  • “Value”: X grid values, can be a single 1D-array (1D) or list or 2 1D-arrays (2D);

  • “Effect”: ALE values corresponding to grid values, can be a single 1D-array (1D) or 2D-array (2D)

• table: DataFrame of ALE results

• options: Dictionary of visualizations configuration for a line (1D numerical) / bar (1D categorical) / heatmap (2D) effect plot. Run results.plot() to show all plots; To display one preferred plot by results.plot(name=xxx), and the following names are available:

  • None: Effect plots of all effects specified in features.

  • “<effect_name>”: Effect plot of the selected main effect or pairwise interaction.

Raises:

ValueError

If attempting 2D ALE plot with categorical features.

Notes

For single features, generates a line or bar plot depending on feature type. For two features, generates a heatmap showing the interaction effects.

Examples

explain_hstatistic(features: Tuple | List = None, dataset: str = 'test', sample_size: int = 5000, percentiles: Tuple = (0, 1), grid_resolution: int = 10, response_method: str = 'auto', random_state: int = 0)

Calculate H-statistics for all feature pairs to measure feature interactions.

The H-statistic measures the strength of interaction effects between pairs of features by comparing their joint effect to the sum of their individual effects. It quantifies how much of the combined effect of two features comes from their interaction.

1. An H-statistic value of 0 indicates no interaction, meaning the features act independently. Values closer to 1 suggest stronger interaction effects between the features.

2. The statistic can be difficult to compare across feature pairs because the denominator varies depending on the pair, and weak main effects can lead to misleadingly high values.

3. Values greater than 1 are possible but harder to interpret, occurring when the variance of the interaction effect exceeds that of the partial dependence plot.

Parameters:

featurestuple, default=None

List of feature names for calculating the H-statistics. If None, all features will be used.

dataset{“main”, “train”, “test”}, default=”test”

Dataset to use for calculating the explanation results.

sample_sizeint, default=5000

Number of random samples to use for speeding up calculation. If None, all data will be used.

percentilesTuple[float, float], default=(0, 1)

Lower and upper percentiles used to create the extreme values for the grid. Must be in [0, 1].

grid_resolutionint, default=10

Number of equally spaced points on the grid for each target feature.

response_method{“auto”, “decision_function”, “predict_proba”}, default=”auto”

Prediction method to use for binary classification tasks:

• “auto”: Uses ‘predict_proba’ if available, otherwise ‘decision_function’

• “predict_proba”: Probability of the positive class

• “decision_function”: Model’s decision function output

random_stateint, default=0

Random seed for controlling randomness in subsampling.

Returns:

ValidationResult

Object containing:

• key: “explain_hstatistic”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the analysis

• value: Dictionary containing:

  • “<feature_name>”: Dictionary of H-statistics between this feature to the rest features.

• table: DataFrame of H-statistics for all feature pairs

• options: Dictionary of visualizations configuration for a horizontal bar plot where x-axis is H-statistics, and y-axis is the feature name. Run results.plot() to show this plot.

Examples

explain_lime(dataset: str = 'test', sample_index: int = 0, centered: bool = True, random_state: int = 0)

Generate a LIME (Local Interpretable Model-agnostic Explanations) explanation for a specific sample.

This method provides local feature importance and contribution analysis for a single prediction using the LIME algorithm. It supports both regression and classification tasks.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

The data set used for calculating the explanation results.

sample_indexint, default=0

The index of the sample in the selected dataset to be explained.

centeredbool, default=True

Whether to center the feature values by subtracting the mean of each feature.

random_stateint, default=0

Random seed for LIME’s perturbation sampling process. Use the same value for reproducible results.

Returns:

ValidationResult

A result object containing:

• key: “explain_lime”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the analysis

• value: Dictionary containing

  • “Name”: List of feature names;

  • “Value”: List of feature values;

  • “Effect”: List of feature contributions measured by LIME;

  • “Coefficient”: List of feature coefficients measured by LIME.

• value: Dictionary with feature names, values, contributions, and coefficients

• table: DataFrame representation of the explanation

• options: Dictionary of visualizations configuration for a horizontal bar plot where x-axis is LIME value, and y-axis is the feature name. Run results.plot() to show this plot.

Notes

The explanation includes both feature coefficients (importance) and feature contributions (coefficient x feature value). Results are visualized using a stem-and-bar plot showing both the feature values and their contributions.

Examples

explain_pdp(features: str | Tuple[str] = None, dataset: str = 'test', sample_size: int = 5000, percentiles: Tuple = (0, 1), grid_resolution: int = 20, response_method: str = 'auto', random_state: int = 0)

Calculate and visualize Partial Dependence Plot (PDP) for specified model features.

Partial Dependence Plots (PDP) show the marginal effect of one or two features on the predicted outcome of a machine learning model. They illustrate how the model’s predictions change as a feature varies over its range, while averaging out the effects of all other features. This makes PDPs a valuable tool for understanding feature importance and their relationships with the target variable in a model-agnostic way.

Parameters:

featuresstr or tuple of str

Name of single feature or tuple of two feature names to analyze their effects on model output.

• If features=(“X1”, ) or “X1”, visualize the main effect for X1.

• If features=(“X1”, “X2”), visualize the interaction for X1 and X2.

• If features=((“X1”, ), (“X2”, )), visualize the main effect for X1 and X2 separately.

Note: Batch mode for 2D effect plot is not supported. If None, all 1D features will be used.

dataset{“main”, “train”, “test”}, default=”test”

The dataset used for calculating the PDP results.

sample_sizeint, default=5000

Number of random samples to use for speeding up calculation. If None, all data points will be used.

percentilestuple, default=(0, 1)

The lower and upper percentile used to create the extreme values for the grid. Must be in [0, 1].

grid_resolutionint, default=20

The number of equally spaced points on the grid for each target feature.

response_method{“auto”, “decision_function”, “predict_proba”}, default=”auto”

Prediction method to use for binary classification tasks:

• “auto”: Uses ‘predict_proba’ if available, otherwise ‘decision_function’

• “predict_proba”: Probability of the positive class

• “decision_function”: Model’s decision function output

random_stateint, default=0

Random seed for controlling reproducibility in subsampling.

Returns:

ValidationResult

PDP result containing:

• key: “explain_pdp”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the analysis

• value: Dictionary containing

  • “Value”: X grid values, can be a single 1D-array (1D) or list or 2 1D-arrays (2D);

  • “Effect”: PD values corresponding to grid values, can be a single 1D-array (1D) or 2D-array (2D)

• table: DataFrame of PDP results

• options: Dictionary of visualizations configuration for a line (1D numerical) / bar (1D categorical) / heatmap (2D) effect plot. Run results.plot() to show all plots; To display one preferred plot by results.plot(name=xxx), and the following names are available:

  • None: Effect plots of all effects specified in features.

  • “<effect_name>”: Effect plot of the selected main effect or pairwise interaction.

Notes

For single features, generates a line or bar plot depending on feature type. For two features, generates a heatmap showing the interaction effects.

Examples

explain_pfi(dataset: str = 'test', sample_size: int = 5000, n_repeats: int = 10, random_state: int = 0)

Calculate Permutation Feature Importance (PFI) for model features.

PFI measures feature importance by calculating the increase in model prediction error after permuting each feature’s values. A feature is considered important if permuting its values increases model error, indicating the model relied on that feature for prediction.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

Dataset to use for calculating the explanation results.

sample_sizeint, default=5000

Maximum number of random samples to use for calculation. If the dataset is larger, a random subset of this size will be used to improve computation speed. Set to None to use the entire dataset.

n_repeatsint, default=10

Number of times to repeat the permutation process. Higher values give more reliable importance estimates but increase computation time.

random_stateint, default=0

Random seed for reproducibility of permutations and sampling.

Returns:

ValidationResult

A result object containing:

• key: “explain_pfi”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the analysis

• value: Dictionary containing:

  • “Name”: List of feature names

  • “Importance”: List of corresponding feature importance values

• table: DataFrame of feature importance results

• options: Dictionary of visualizations configuration for a horizontal bar plot where x-axis is permutation feature importance, and y-axis is the feature name. Run results.plot() to show this plot.

Examples

explain_shap(dataset: str = 'test', sample_index: int = 0, baseline_dataset: str = 'train', baseline_sample_index: int = None, baseline_sample_size: int = 500, random_state: int = 0)

Generate SHAP (SHapley Additive exPlanations) values for local model explanation.

This method uses either Kernel SHAP or Baseline SHAP to explain model predictions for a specific sample. If baseline_sample_index is provided, Baseline SHAP is used with the specified reference point. Otherwise, Kernel SHAP is used with randomly sampled background data.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

Dataset containing the sample to be explained.

sample_indexint, default=0

Index of the sample to be explained within the selected dataset.

baseline_dataset{“main”, “train”, “test”}, default=”train”

Dataset to use for background/reference data.

baseline_sample_indexint, optional

Index of the baseline sample to use as reference point. If None, random samples will be drawn as background data.

baseline_sample_sizeint, default=500

Number of background samples to use when baseline_sample_index is None. Controls computation speed vs. accuracy trade-off.

random_stateint, default=0

Random seed for reproducible background sampling.

Returns:

ValidationResult

A result object containing:

• key: “explain_shap”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the analysis

• value: Dictionary containing

  • “Name”: List of feature names;

  • “Value”: List of feature values;

  • “Effect”: List of feature contributions measured by SHAP.

• value: Contains the core numerical results (feature names, values, and SHAP contributions)

• table: Provides a tabular summary combining all the information

• options: Dictionary of visualizations configuration for a horizontal bar plot where x-axis is SHAP value, and y-axis is the feature name. Run results.plot() to show this plot.

Notes

For classification tasks, SHAP values are computed for the positive class (class 1) probability. For regression tasks, SHAP values are computed for the predicted value.

Examples

export_report(path: str = 'report.html')

Export report to html

Parameters:

pathstr, optional

The export path, by default “report.html”

get_dataset()

Return the dataset object.

get_interactions()

Return the list of pairwise interactions.

Only available for FANOVA models.

Returns:

The list of 2-way interactions.

get_main_effects()

Return the list of main effects.

Only available for FANOVA models.

Returns:

The list of main effects.

get_model()

Return the model object.

interpret_coef(features: str | Tuple[str] = None)

Extracts and visualizes the coefficients of linear model features.

This function retrieves the coefficients from the model’s main effects and creates both a tabular and graphical representation of the coefficients. The coefficients can be filtered to show only specific features of interest.

Parameters:

featuresstr or tuple of str, default=None

Specific features whose coefficients should be displayed. If None, coefficients for all features will be shown. Can be either a single feature name (str) or multiple feature names (tuple).

Returns:

ValidationResult

A result object containing:

• key: “interpret_glm_coef”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters

• value: Dictionary containing:

  • “Name”: List of feature names

  • “Coefficients”: List of corresponding feature coefficients

• table: pandas DataFrame of feature names and coefficients

• options: Dictionary of visualizations configuration for a horizontal bar plot where x-axis is coefficients, and y-axis is the feature names. Run results.plot() to show this plot.

Examples

interpret_effects(features: str | Tuple = None, dataset: str = 'test', grid_size: int = 200)

Analyze and visualize how one or two features influence model predictions through main effects or interaction effects.

This method computes and visualizes how specified features affect the model’s output, allowing for both single and dual feature analysis. It generates visualizations such as line plots, bar plots, or heatmaps based on the type of features provided, and returns a structured result encapsulating the effects and visual options.

Parameters:

featuresstr or tuple of str

Name of single feature or tuple of two feature names to analyze their effects on model output.

• If features=(“X1”, ) or “X1”, visualize the main effect for X1.

• If features=(“X1”, “X2”), visualize the interaction for X1 and X2.

• If features=((“X1”, ), (“X2”, )), visualize the main effect for X1 and X2 separately.

Note: Batch mode for 2D effect plot is not supported. If None, all 1D features will be used.

dataset{“main”, “train”, “test”}, default=”test”

Dataset partition to use for effect calculations. Controls which data points are used to determine the range and importance of effects.

grid_sizeint, default=200

Resolution of the effect visualization grid. Higher values create smoother visualizations but increase computation time.

Returns:

ValidationResult

A container object with the following components:

• key: “interpret_effect”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters

• value: Dictionary containing:

  • “Value”: Feature values used for effect calculation

  • “Effect”: Calculated effect values

  • “Details”: A dict containing details about the effects

• table: DataFrame containing the effect results

  • For single feature: columns are “Value” and “Effect”

  • For two features: 2D table with features as row/column indices

• options: Dictionary of visualizations configuration for a line (1D numerical) / bar (1D categorical) / heatmap (2D) effect plot. Run results.plot() to show all plots; To display one preferred plot by results.plot(name=xxx), and the following names are available:

  • None: Effect plots of all effects specified in features.

  • “<effect_name>”: Effect plot of the selected main effect or pairwise interaction.

Examples

interpret_effects_moe_average(features: str | Tuple, dataset: str = 'test', grid_size: int = 100, sample_size: int = 5000, random_state: int = 0)

Analyze feature effects averaged across all mixture-of-experts clusters.

Parameters:

featuresstr or tuple/list of str

One or two feature names to analyze. If two features are provided, their interaction effect is analyzed.

dataset{“main”, “train”, “test”}, default=”test”

The dataset to use for the analysis.

grid_sizeint, default=100

Number of points to evaluate for creating the visualization grid.

sample_sizeint, default=5000

Maximum number of random samples to use for calculation efficiency. If None, uses all data.

random_stateint, default=0

Random seed for reproducible sampling.

Returns:

ValidationResult

Contains averaged effect analysis results:

• key: “interpret_effects_moe_average”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters

• value: Weighted average effect values

• table: Tabular representation of results

• options: Dictionary of visualizations configuration for a line (1D numerical) / bar (1D categorical) / heatmap (2D) averaged effect plot. Run results.plot() to show this plot.

Examples

interpret_ei(dataset: str = 'test')

Calculate and visualize global effect importance for model features.

This function computes the global effect importance of features in a specified dataset and generates a visualization of the importance values. It retrieves the feature data, calculates their importance, and returns a structured result containing the feature names, their importance values, and a visualization.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

Dataset partition to use for calculating effect importance. Controls which data points are used in the analysis.

Returns:

ValidationResult

A container object with the following components:

• key: “interpret_ei”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters

• value: Dictionary containing:

  • “Name”: Names of the features/effects

  • “Importance”: Corresponding importance values

• table: DataFrame with feature names and importance values

• options: Dictionary of visualizations configuration for a horizontal bar plot where x-axis is importance, and y-axis is the effect names. Run results.plot() to show this plot.

Examples

interpret_fi(dataset: str = 'test')

Calculate and visualize global feature importance for the model.

This function computes the importance of each feature in the model’s predictions and creates a horizontal bar plot visualization of the results.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

The data set used for calculating the explanation results.

Returns:

ValidationResult

A container object with the following components:

• key: “interpret_ei”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters

• value: Dictionary containing:

  • “Name”: List of feature names

  • “Importance”: List of corresponding feature importance values

• table: DataFrame containing feature names and importance values

• options: Dictionary of visualizations configuration for a horizontal bar plot where x-axis is importance, and y-axis is the feature names. Run results.plot() to show this plot.

Examples

interpret_global_tree()

Generate a visualization of the complete decision tree model structure.

This method creates a tree diagram showing all nodes and splits in the decision tree model, providing a global view of the model’s decision-making structure.

Returns:

ValidationResult

A result object containing:

• key: “interpret_tree_global”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters

• options: Dictionary of visualizations configuration for the global tree diagram. Run results.plot() to show this plot.

Examples

interpret_llm_pc(dataset: str = 'test')

Generate and visualize parallel coordinate plots for Local Linear Model (LLM) coefficients.

This function calculates LLM statistics, feature importance, and creates a parallel coordinate plot to visualize the relationships between features and their coefficients across different local linear models.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

The data set used for calculating the explanation results.

Returns:

ValidationResult

A container object with the following components:

• key: “llm_pc”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters

• value: Dictionary containing:

  • “Names”: Feature names

  • “Bias & Weights”: LLM coefficients including bias terms

  • “STD_DEV”: Standard deviations of LLM predictions

  • “Importance”: Feature importance scores

• options: Dictionary of visualizations configuration for a parallel coordinate plot where x-axis is feature names, and y-axis is the local linear coefficients. Run results.plot() to show this plot.

Examples

interpret_llm_profile(feature: str = None, dataset: str = 'test')

Calculate local feature importance for a specific feature using LLM profiles.

This function computes the local feature importance for a given feature by analyzing the LLM profiles and visualizes the results. It retrieves the necessary data, calculates the feature importance, and generates a plot that illustrates the distribution of feature importance across different LLMs.

Parameters:

featurestr

Feature name to explain.

dataset{“main”, “train”, “test”}, default=”test”

The data set used for calculating the explanation results.

nllmsint, default=30

The number of top LLMs to show.

Returns:

ValidationResult

A container object with the following components:

• key: “llm_profile”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters

• value: Dictionary containing:

  • “feature”: Name of the feature being analyzed

  • “sample_idx_by_llms”: Sample indices grouped by LLMs

  • “feature_names”: List of feature names

  • “feature_importance”: Feature importance scores

• options: Dictionary of visualizations configuration for a LLM profile plot. Run results.plot() to show this plot.

Examples

interpret_llm_summary(dataset: str = 'test')

This method generates a table populated with unwrapper summary statistics.

The statistics include count, response mean, response std deviation, local MSE/AUC and global MSE/AUC for selected top LLMs

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

The data set used for calculating the explanation results.

Returns:

ValidationResult

A container object with the following components:

• key: “llm_summary”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters

• table: DataFrame containing summary statistics including:

  • “count”

  • “response mean”

  • “response standard deviation”

  • “local MSE/AUC”

  • “global MSE/AUC”

Examples

interpret_llm_violin(feature: str = None, dataset: str = 'test')

Generates the LLM coefficients and statistics. (Violin plot under development)

This function processes the specified dataset to calculate the LLM coefficients and their associated statistics, ultimately returning a ValidationResult object that encapsulates the results of the analysis, including feature names, importance scores, and bias weights.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

The data set used for calculating the explanation results.

Returns:

ValidationResult

A container object with the following components:

• key: “llm_violin”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters

• value: Dictionary containing:

  • “feature_names”: List of feature names

  • “all_bias_weight”: LLM coefficients including bias terms

  • “count_llms”: Count of LLMs

  • “feature_importance”: Feature importance scores

Examples

interpret_local_ei(dataset: str = 'test', sample_index: int = 0)

Calculate and visualize local effect importance for a specific data sample.

This function computes the importance of different features’ effects on the model’s prediction for a single instance and visualizes the results.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

The data set used for calculating the explanation results.

sample_indexint, default=0

Index of the specific sample in the selected dataset to analyze.

Returns:

ValidationResult

A container object with the following components:

• key: “interpret_local_ei”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters

• value: Dictionary containing:

  • “Name”: List of feature names

  • “Importance”: Feature importance scores

  • “Values”: Feature values for the selected sample

• table: DataFrame containing feature names and importance values

• options: Dictionary of visualizations configuration for a horizontal bar-stem plot where x-axis is importance, and y-axis is the feature names. Run results.plot() to show this plot.

Examples

interpret_local_fi(dataset: str = 'test', sample_index: int = 0, centered: bool = True)

Calculates and visualizes feature importance scores for a single sample.

This function computes the local feature importance scores for a specified sample from the dataset, visualizes the results, and returns a structured validation result containing the analysis details.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

The data set used for calculating the explanation results.

sample_indexint, default=0

Index of the specific sample to analyze within the selected dataset

centeredbool, default=True

Whether to center the feature importance scores by subtracting the mean effect of each feature across all samples

Returns:

ValidationResult

A container object with the following components:

• key: “interpret_fi_local”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the analysis

• value: Dictionary containing:

  • “Name”: List of feature names

  • “Importance”: Feature importance scores

  • “Values”: Original feature values

• table: DataFrame version of the value dictionary

• options: Dictionary of visualizations configuration for a horizontal bar-stem plot where x-axis is importance, and y-axis is the feature names. Run results.plot() to show this plot.

Examples

interpret_local_linear_fi(dataset: str = 'test', sample_index: int = 0, centered: bool = True)

Calculate and visualize local feature importance for a specific data sample using linear approximation.

This function computes the local feature importance scores for a given sample from the specified dataset using a linear model. It visualizes the importance scores alongside the original feature values and coefficients, providing insights into the contribution of each feature to the model’s prediction.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

Specifies which dataset partition to use for the analysis.

sample_indexint, default=0

Index of the specific sample in the selected dataset to analyze.

centeredbool, default=True

If True, features are centered by subtracting their mean values before calculating importance scores.

Returns:

ValidationResult

A container object with the following components:

• key: “interpret_local_linear_fi”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters

• value: Dictionary containing:

  • “Name”: List of feature names

  • “Importance”: Feature importance scores

  • “Values”: Original feature values

  • “Coefficients”: Linear coefficients

• table: DataFrame containing feature names, scores, values, and coefficients

• options: Dictionary of visualizations configuration for a horizontal bar-stem plot where x-axis is importance, and y-axis is the feature names. Run results.plot() to show this plot.

Examples

interpret_local_moe_weights(dataset: str = 'test', sample_index: int = 0)

Calculate and visualize expert weights for a specific sample.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

The dataset containing the sample to analyze.

sample_indexint, default=0

The index of the sample in the selected dataset.

Returns:

ValidationResult

Contains mixture-of-experts weight analysis:

• key: “interpret_local_moe_weights”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters

• value: Dictionary with expert names, weights, feature names and values

• options: Dictionary of visualizations configuration for a bar plot where x-axis is expert id, and y-axis is the expert weights. Run results.plot() to show this plot.

Examples

interpret_local_tree(dataset: str = 'test', sample_index: int = 0)

Generate a visualization of the decision path for a specific sample through the decision tree.

This method creates a tree diagram highlighting the specific nodes and path traversed when classifying/predicting a single sample.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

The data set used for calculating the explanation results.

sample_indexint, default=0

The index of the sample in the selected dataset to visualize the decision path for.

Returns:

ValidationResult

A result object containing:

• key: “interpret_tree_local”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters

• options: Dictionary of visualizations configuration for the local tree diagram. Run results.plot() to show this plot.

Examples

interpret_moe_cluster_analysis(dataset: str = 'test', metric: str = None)

Analyze and summarize characteristics of mixture-of-experts clusters.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

The dataset to analyze cluster assignments.

metricstr, metric=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

Returns:

ValidationResult

Contains cluster analysis results:

• key: “interpret_cluster_analysis”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters

• value: Nested dictionary containing the (“<expert_id>”, item) pairs for each group, and the item is also a dictionary with:

  • “size”: Number of samples in cluster

  • “score”: The performance metric of this cluster

  • “center”: Cluster centroid coordinates

  • “data_info”: Sample indices for in/out of cluster comparison, which can be further used for data distribution test, e.g.,

    data_results = ds.data_drift_test(**results.value[2]["data_info"])
    data_results.plot("summary")
    data_results.plot(("density", "MedInc"))
    

• table: DataFrame with performance metrics for each cluster

• options: Dictionary of visualizations configuration. Run results.plot(name=xxx) to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

  • “cluster_performance”: Bar plot visualizing the performance scores of final MOE model against each cluster.

Examples

list_registered_tests(name: str = None)

Return the list all registered tests.

Parameters:

namestr

The name of test used for filtering.

load_registered_test(name: str, run_id: str = None)

Load config and result of registered tests.

Parameters:

namestr

The name of test used for filtering.

run_idstr, default=None

run id of the registered test.

register(name: str, test_result: ValidationResult, register_name: str = None, description: str = None, tags: Dict[str, Any] | None = None, run_id: str = None)

Register a test into MLFlow.

Parameters:

namestr

The current name of the test to be registered.

test_resultValidationResult

The validation result object of test.

register_namestr, default=None

The register name of the test in MLFlow. If None, will be the same as name.

descriptionstr, default=None

The description of this test.

tagsdict, default=None

The tags.

run_idstr, default=None

The run id in MLFLow.

set_dataset(dataset: LocalDataSet)

Set dataset for test suite.

Parameters:

datasetDataset

Dataset object

set_model(model: ModelBase)

Set model for test suite.

Parameters:

modelModelBase

modeva built-in models or external models that implement ModelBase