Model Comparison

Side-by-side comparison of multiple registered models across the diagnostic areas, exposed by TestSuite.

TestSuite.compare_accuracy_table(train_dataset: str = 'train', test_dataset: str = 'test', metric: str | Tuple = None)

Compare predictive performance metrics across multiple models.

This function evaluates and compares the predictive performance of multiple models using specified metrics on both training and test datasets.

Parameters:

train_datasetstr, default=”train”

Specifies the training dataset to evaluate. Options: “main”, “train”, or “test”

test_datasetstr, default=”test”

Specifies the test dataset to evaluate. Options: “main”, “train”, or “test”

metricstr or tuple of str, default=None

Performance metric(s) to calculate. If None: - For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier” - For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

Returns:

ValidationResult

A container object with the following components:

• key: “compare_accuracy_table”

• data: Name of the dataset used

• model: List of model names being compared

• inputs: Input parameters used for the comparison

• value: Dictionary of (“<model_name>”, item), each item is also a nested dictionary with (“<metric_name>”, subitem), where each subitem is also a dictionary with:

  • “<train_dataset>”: The metric value of training dataset.

  • “<test_dataset>”: The metric value of testing dataset.

  • “GAP”: The performance gap is calculated as (test_score - train_score).

• table: Pandas DataFrame containing detailed performance metrics

• value: Dictionary of (“<model_name>”, item), each item is also a nested dictionary with (“<metric_name>”, subitem), where each subitem is also a dictionary with:

  • “<train_dataset>”: The metric value of training dataset.

  • “<test_dataset>”: The metric value of testing dataset.

  • “GAP”: The performance gap is calculated as (test_score - train_score).

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

  • “<metric_name>”: a bar plot where x-axis is the model names, and y-axis is performance metric

Examples

TestSuite.compare_slicing_accuracy(features: str, dataset: str = 'test', metric: str = None, method: str = 'uniform', bins: int | Dict = 10, n_estimators: int = 1000, threshold: float | int = None)

Compares model performance across different data slices based on a specified feature.

This function evaluates the performance of multiple models on different slices of the dataset, allowing for a detailed comparison based on a specified feature. It computes performance metrics for each model across the defined slices and returns a structured result containing the analysis.

Parameters:

featuresstr

Name of the feature to use for data slicing.

dataset{“main”, “train”, “test”}, default=”test”

The data set to be tested.

metricstr, default=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

method{“uniform”, “quantile”, “auto-xgb1”, “precompute”}, default=”uniform”

Method for binning numerical features:

• “uniform”: Equal-width binning

• “quantile”: Equal-frequency binning

• “auto-xgb1”: XGBoost-based automatic binning

• “precompute”: Use pre-specified bin edges

binsint or dict, default=10

Controls binning granularity:

• If int: Number of bins for numerical features. For “quantile”, this is the maximum number of bins. For “auto-xgb1”, this sets XGBoost’s max_bin parameter.

• If dict: Manual bin specifications for each feature, only used with method=”precompute”. Format: {feature_name: array_of_bin_edges}. Example: {“X0”: [0.1, 0.5, 0.9]} Note: Cannot specify bins for categorical features.

n_estimatorsint, default=1000

Number of estimators for XGBoost when method=”auto-xgb1”

thresholdfloat or int, default=None

Threshold for filtering fairness metric results. If not specified, it will not be used.

Returns:

ValidationResult

A container object with the following components:

• key: “compare_slicing_accuracy”

• data: Name of the dataset used

• model: List of model names being compared

• inputs: Input parameters used for the analysis

• value: Dictionary of (“<model_name>”, item) pairs, where each item is a list of dict, containing the information about the performance metrics for each segment,

  • “Feature”: feature name

  • “Segment”: segment value (categorical) or segment range (numerical)

  • “Size”: number of samples in this segment

  • <”metric”>: performance metric value of this segment

  • “Sample_ID”: sample indices of this segment

  • “Sample_Dataset”: dataset name, e.g., “train”, “test”, etc.

  • “Segment_Info”: explicit definition of this segment, similar to “Segment”

• table: Pandas DataFrame with detailed slicing statistics

• options: Dictionary of visualizations configuration for a mulit-line plot where x-axis is the selected slicing feature, and y-axis is performance metric. Run results.plot() to show this plot.

Examples

TestSuite.compare_slicing_overfit(features: str, train_dataset: str = 'train', test_dataset: str = 'test', metric: str = None, method: str = 'uniform', bins: int | Dict = 10, n_estimators: int = 1000, threshold: float | int = None)

Compares model performance across different data slices to identify potential overfit regions.

This function evaluates the performance of multiple models on specified data slices, allowing for the identification of regions where a model may be overfitting. It utilizes various binning methods to segment the data based on the specified feature and computes performance metrics to assess the differences between training and testing datasets.

Parameters:

featuresstr

Name of the feature to use for data slicing.

train_dataset{“main”, “train”, “test”}, default=”train”

Specifies which dataset to use as the training set for comparison.

test_dataset{“main”, “train”, “test”}, default=”test”

Specifies which dataset to use as the test set for comparison.

metricstr, default=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

method{“uniform”, “quantile”, “auto-xgb1”, “precompute”}, default=”uniform”

Method for binning numerical features:

• “uniform”: Equal-width binning

• “quantile”: Equal-frequency binning

• “auto-xgb1”: XGBoost-based automatic binning

• “precompute”: Use pre-specified bin edges

binsint or dict, default=10

Controls binning granularity:

• If int: Number of bins for numerical features. For “quantile”, this is the maximum number of bins. For “auto-xgb1”, this sets XGBoost’s max_bin parameter.

• If dict: Manual bin specifications for each feature, only used with method=”precompute”. Format: {feature_name: array_of_bin_edges}. Example: {“X0”: [0.1, 0.5, 0.9]} Note: Cannot specify bins for categorical features.

n_estimatorsint, default=1000

Number of trees for XGBoost when using method=”auto-xgb1”

thresholdfloat or int, default=None

Threshold for filtering overfit regions. If not specified, it will not be used.

Returns:

ValidationResult

A container object with the following components:

• key: “compare_slicing_overfit”

• data: Name of the dataset used

• model: List of model names being compared

• inputs: Input parameters used for the analysis

• value: Dictionary of (“<model_name>”, item) pairs, where each item is a nested dictionary with dictionary containing the information about the performance metric gap for each segment,

  • “Feature”: feature name

  • “Segment”: segment value (categorical) or segment range (numerical)

  • “Size”: number of samples in this segment

  • <”metric”>: performance metric gap value of this segment

  • “Sample_ID”: sample indices of this segment

  • “Sample_Dataset”: dataset name, e.g., “train”, “test”, etc.

  • “Segment_Info”: explicit definition of this segment, similar to “Segment”

  • “Weak”: boolean indicator showing whether this segment is weak or not

• table: DataFrame with detailed performance metrics for each slice

• options: Dictionary of visualizations configuration for a mulit-line plot where x-axis is the selected slicing feature, and y-axis is performance metric gap. Run results.plot() to show this plot.

Examples

TestSuite.compare_slicing_reliability(features: str, train_dataset: str = 'test', test_dataset: str = 'test', test_size: float = 0.5, method: str = 'uniform', bins: int | Dict = 10, n_estimators: int = 1000, threshold: float | int = None, metric: str = 'width', alpha: float = 0.1, max_depth: int = 5, random_state: int = 0)

Compares reliability metrics across different slices of data for multiple models.

This function compares reliability metrics, such as width and coverage, across various slices of data for multiple models. It utilizes the specified features to segment the dataset and computes the reliability metrics based on the chosen method and parameters.

Parameters:

featuresstr

Name of the feature to use for slicing the data.

train_dataset{“main”, “train”, “test”}, default=”test”

Dataset to use for training and calibration purposes.

test_dataset{“main”, “train”, “test”}, default=”test”

Dataset to use for evaluation and comparison.

test_sizefloat, default=0.5

Proportion of data to use as test set when train_dataset equals test_dataset. Must be between 0 and 1.

method{“uniform”, “quantile”, “auto-xgb1”, “precompute”}, default=”uniform”

Method to use for creating bins:

• “uniform”: Equal-width bins

• “quantile”: Equal-frequency bins

• “auto-xgb1”: XGBoost-based automatic binning

• “precompute”: Use pre-defined bin edges

binsint or dict, default=10

Controls binning granularity:

• If int: Number of bins for numerical features. For “quantile”, this is the maximum number of bins. For “auto-xgb1”, this sets XGBoost’s max_bin parameter.

• If dict: Manual bin specifications for each feature, only used with method=”precompute”. Format: {feature_name: array_of_bin_edges}. Example: {“X0”: [0.1, 0.5, 0.9]} Note: Cannot specify bins for categorical features.

n_estimatorsint, default=1000

Number of trees for XGBoost when using method=”auto-xgb1”.

thresholdfloat or int, default=None

Threshold for filtering fairness unreliable regions. If not specified, it will not be used.

metric{“width”, “coverage”}, default=”width”

Reliability metric to compute:

• “width”: Average prediction interval width

• “coverage”: Average prediction coverage rate

alphafloat, default=0.1

Target coverage level for prediction intervals (between 0 and 1).

max_depthint, default=5

Maximum tree depth for gradient boosting model (regression tasks only).

random_stateint, default=0

Random seed for reproducibility.

Returns:

ValidationResult

A container object with the following components:

• key: “compare_slicing_reliability”

• data: Name of the dataset used

• model: List of model names being compared

• inputs: Input parameters used for the analysis

• value: Dictionary of (“<model_name>”, item) pairs, where each item is a nested dictionary with dictionary containing the information about the reliability metric for each segment.

  • “Feature”: feature name

  • “Segment”: segment value (categorical) or segment range (numerical)

  • “Size”: number of samples in this segment

  • <”metric”>: reliability metric value of this segment

  • “Sample_ID”: sample indices of this segment

  • “Sample_Dataset”: dataset name, e.g., “train”, “test”, etc.

  • “Segment_Info”: explicit definition of this segment, similar to “Segment”

  • “Weak”: boolean indicator showing whether this segment is weak or not

• table: DataFrame with detailed reliability statistics per slice

• options: Dictionary of visualizations configuration for a mulit-line plot where x-axis is the selected slicing feature, and y-axis is performance metric gap. Run results.plot() to show this plot.

Examples

TestSuite.compare_slicing_robustness(features: str, dataset: str = 'test', metric: str = None, method: str = 'uniform', bins: int | Dict = 10, n_estimators: int = 1000, threshold: float | int = None, n_repeats: int = 10, perturb_features: str | Tuple = None, perturb_method: str = 'normal', noise_levels: float | int = 0.1, random_state: int = 0)

Compares model robustness across different data slices by analyzing performance stability under perturbations.

This function evaluates the robustness of a model by analyzing its performance across various data slices, applying perturbations to the specified features, and measuring the stability of the model’s predictions. It allows for different binning methods and metrics to be used, providing flexibility in how the analysis is conducted.

Parameters:

featuresstr

Name of the feature to use for data slicing.

dataset{“main”, “train”, “test”}, default=”test”

Dataset to analyze.

method{“uniform”, “quantile”, “auto-xgb1”, “precompute”}, default=”uniform”

Method for binning numerical features:

• “uniform”: Equal-width bins

• “quantile”: Equal-frequency bins

• “auto-xgb1”: XGBoost-based automatic binning

• “precompute”: Use pre-defined bin edges

binsint or dict, default=10

Controls binning granularity:

• If int: Number of bins for numerical features. For “quantile”, this is the maximum number of bins. For “auto-xgb1”, this sets XGBoost’s max_bin parameter.

• If dict: Manual bin specifications for each feature, only used with method=”precompute”. Format: {feature_name: array_of_bin_edges}. Example: {“X0”: [0.1, 0.5, 0.9]} Note: Cannot specify bins for categorical features.

metricstr, default=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

n_estimatorsint, default=1000

The number of estimators in xgboost, used when method=”auto-xgb1”.

thresholdfloat or int, default=None

Threshold for filtering fairness non-robust regions. If not specified, it will not be used.

n_repeatsint, default=10

Number of times to repeat perturbation analysis.

perturb_featuresstr or tuple, default=None

Features to perturb. If None, all features are perturbed.

perturb_method{“normal”, “quantile”}, default=”normal”

Method for perturbing numerical features.

noise_levelsfloat or int, default=0.1

Magnitude of perturbation to apply.

random_stateint, default=0

Random seed for reproducibility.

Returns:

ValidationResult

A container object with the following components:

• key: “compare_slicing_robustness”

• data: Name of the dataset used

• model: List of model names being compared

• inputs: Input parameters used for the analysis

• value: Dictionary of (“<model_name>”, item) pairs, where each item is a nested dictionary with dictionary containing the information about the performance metric (after perturbation) for each segment.

  • “Feature”: feature name

  • “Segment”: segment value (categorical) or segment range (numerical)

  • “Size”: number of samples in this segment

  • <”metric”>: perturbed model performance metric value of this segment

  • “Sample_ID”: sample indices of this segment

  • “Sample_Dataset”: dataset name, e.g., “train”, “test”, etc.

  • “Segment_Info”: explicit definition of this segment, similar to “Segment”

  • “Weak”: boolean indicator showing whether this segment is weak or not

• table: DataFrame with comparative performance metrics

• options: Dictionary of visualizations configuration for a mulit-line plot where x-axis is the selected slicing feature, and y-axis is performance metric (after perturbation). Run results.plot() to show this plot.

Examples

TestSuite.compare_slicing_fairness(group_config, features: str, favorable_label: int = 1, dataset: str = 'test', metric: str = None, method: str = 'uniform', bins: int | Dict = 10, n_estimators: int = 1000, threshold: float | int = None)

Evaluates fairness metrics across different protected and reference groups by slicing the data.

This function computes fairness metrics for specified groups in the dataset by analyzing the model’s predictions against the actual outcomes, allowing for a detailed comparison of fairness across different slices of the data.

Parameters:

group_configdict

Configuration defining protected and reference groups. Each key is a custom group name, and each value is a dictionary with group definitions. Supports three formats:

1. For numerical features:

   {

   “feature”: str, # Feature name “protected”: { # Protected group bounds

   “lower”: float, # Lower bound “lower_inclusive”: bool, “upper”: float, # Optional upper bound “upper_inclusive”: bool

   }, “reference”: { # Reference group bounds

   “lower”: float, # Optional lower bound “lower_inclusive”: bool, “upper”: float, # Upper bound “upper_inclusive”: bool

   }

   }

2. For categorical features:

   {

   “feature”: str, # Feature name “protected”: str or int, # Protected group category “reference”: str or int # Reference group category

   }

3. For probabilistic group membership:

   {

   “by_weights”: True, “protected”: str, # Column name with protected group probabilities “reference”: str # Column name with reference group probabilities

   }

featuresstr

Name of the feature to use for slicing the data

favorable_label{0, 1}, default=1

For classification: The preferred class label. For regression: 1 means larger predictions are preferred, 0 means smaller predictions are preferred.

dataset{“main”, “train”, “test”}, default=”test”

Which dataset partition to analyze

metricstr, default=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

method{“uniform”, “quantile”, “auto-xgb1”, “precompute”}, default=”uniform”

Method for binning numerical features

binsint or dict, default=10

Controls binning granularity:

• If int: Number of bins for numerical features. For “quantile”, this is the maximum number of bins. For “auto-xgb1”, this sets XGBoost’s max_bin parameter.

• If dict: Manual bin specifications for each feature, only used with method=”precompute”. Format: {feature_name: array_of_bin_edges}. Example: {“X0”: [0.1, 0.5, 0.9]} Note: Cannot specify bins for categorical features.

n_estimatorsint, default=1000

Number of estimators in xgboost, used when method=”auto-xgb1”.

thresholdfloat or int, default=None

Threshold for filtering fairness metric results. If not specified, it will not be used.

Returns:

ValidationResult

A container object with the following components:

• key: “compare_slicing_fairness”

• data: Name of the dataset used

• model: List of model names analyzed

• inputs: Input parameters used

• value: Dictionary of (“<model_name>”, item) pairs, and the item is also a dictionary with:

  • “<group_name>”: List of fairness metrics for each segment, and each element is a dict containing

    • “Feature”: feature name

    • “Segment”: segment value (categorical) or segment range (numerical)

    • “Size”: number of samples in this segment

    • <”metric”>: fairness metric value of this segment

    • “Sample_ID”: sample indices of this segment

    • “Sample_Dataset”: dataset name, e.g., “train”, “test”, etc.

    • “Segment_Info”: explicit definition of this segment, similar to “Segment”

    • “Weak”: boolean indicator showing whether this segment is weak or not

• table: dictionary of fairness metric table.

  • “<group_name>”: Table of fairness metrics for each segment.

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

  • “<group_name>”: Line plots visualizing the fairness metrics against the slicing feature.

Examples

TestSuite.compare_robustness(dataset: str = 'test', metric: str = None, n_repeats: int = 10, perturb_features: str | Tuple = None, perturb_method: str = 'normal', noise_levels: float | int | Tuple = 0.1, random_state: int = 0)

Performs robustness testing by comparing model performance under different perturbation levels.

This function performs robustness testing by comparing the performance of different models when subjected to various levels of perturbation. It allows for the specification of the dataset, performance metric, and perturbation characteristics, and returns a comprehensive result encapsulating the robustness scores and visualizations.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

Dataset partition to be used for testing.

metricstr, default=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

n_repeatsint, default=10

Number of times to repeat the perturbation test.

perturb_featuresstr or tuple, default=None

Specific features to perturb. If None, all features are perturbed.

perturb_method{“normal”, “quantile”}, default=”normal”

Method to generate perturbations:

• “normal”: Gaussian noise

• “quantile”: Quantile-based perturbation

noise_levelsfloat or tuple, default=0.1

Magnitude of perturbation to apply. Can be single value or multiple levels.

random_stateint, default=0

Seed for random number generation to ensure reproducibility.

Returns:

ValidationResult

A container object with the following components:

• key: “compare_robustness”

• data: Name of the dataset used

• model: List of model names being compared

• inputs: Input parameters used for the test

• value: Dictionary of (“<model_name>”, item) pairs, which item is also a dictionary with

  • noise_level (e.g., 0.1, 0.2): Performance metrics across noise levels;

  • data_info: Sample indices for worst and remaining cases (only for the first noise level)

    data_results = ds.data_drift_test(**results.value["MoLGBMRegressor"][0.2]["data_info"])
    data_results.plot("summary")
    data_results.plot(("density", "MedInc"))
    

• table: DataFrame containing detailed performance metrics for each noise level

• options: Dictionary of visualizations configuration for a bar plot visualizing the performance scores against each cluster. Run results.plot() to show this plot.

Examples

TestSuite.compare_reliability(train_dataset: str = 'test', test_dataset: str = 'test', test_size: float = 0.5, alpha: float = 0.1, max_depth: int = 5, random_state: int = 0)

Compares reliability performance of multiple models under data shifts by evaluating prediction intervals/sets.

This function evaluates the prediction intervals/sets of various models by comparing their reliability metrics on specified training and testing datasets. It aggregates results such as average width and coverage for each model, providing insights into their performance under different conditions.

Parameters:

train_dataset{“main”, “train”, “test”}, default=”test”

Dataset used for model training and calibration. Choose from available dataset splits.

test_dataset{“main”, “train”, “test”}, default=”test”

Dataset used for evaluation. Choose from available dataset splits.

test_sizefloat, default=0.5

Proportion of data to use as test set when train_dataset equals test_dataset. Must be between 0 and 1.

alphafloat, default=0.1

Target miscoverage rate for prediction intervals/sets. Must be between 0 and 1.

max_depthint, default=5

Maximum depth of the GBM trees used for quantile regression. Only applicable for regression tasks.

random_stateint, default=0

Random seed for reproducible results.

Returns:

ValidationResult

A container object with the following components:

• key: “compare_reliability”

• data: Name of the dataset used

• model: List of model names being compared

• inputs: Input parameters used for the analysis

• value: Dictionary of (“<model_name>”, item), each item is also a dictionary with:

  • “interval”: Prediction intervals / sets and related metrics

  • “data_info”: The sample indices of reliable and unreliable samples, which can be further used for data distribution test, e.g.,

    data_results = ds.data_drift_test(**results.value["LGBMRegressor"]["data_info"])
    data_results.plot("summary")
    data_results.plot(("density", "MedInc"))
    

• table: DataFrame with detailed reliability metrics including average width and coverage for each model

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

  • width: Bar plot comparing empirical widths across models

  • coverage: Bar plot comparing empirical coverage across models

Examples

TestSuite.compare_resilience(dataset: str = 'test', method: str = 'worst-sample', metric: str = None, alphas: tuple = None, n_clusters: int = 10, random_state: int = 0)

Compare model resilience performance under data shifts across multiple models.

This function compares the performance of different models under data shifts by evaluating their resilience scores. It allows users to specify the dataset partition, performance metric, and method for identifying problematic samples, and it returns a comprehensive result encapsulating the resilience scores and performance metrics.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

The dataset partition to analyze.

metricstr, default=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

method{“worst-sample”, “worst-cluster”, “outer-sample”, “hard-sample”}, default=”worst-sample”

Strategy for identifying challenging samples:

• “worst-sample”: Ranks samples by their prediction error

• “worst-cluster”: Groups samples into clusters and identifies problematic clusters

• “outer-sample”: Uses PCA to detect statistical outliers

• “hard-sample”: Trains a metamodel to identify inherently difficult samples

alphastuple of float, default=None

Fraction of worst ratios within (0, 1]. If None, it defaults to (0.1, 0.2, 0.3, …, 0.9, 1.0).

n_clustersint, default=10

Number of clusters when using method=”worst-cluster”.

random_stateint, default=0

Random seed for reproducibility.

Returns:

ValidationResult

A container object with the following components:

• key: “compare_resilience”

• data: Name of the dataset used

• model: List of model names being compared

• inputs: Input parameters used

• value: Dictionary of (“<model_name>”, item) pairs, which item is also a dictionary with scores for different fractions of worst samples (0.1 to 1.0) and corresponding sample indices.

  • “interval”: Prediction intervals / sets and related metrics

  • “data_info”: The sample indices of reliable and unreliable samples, which can be further used for data distribution test, e.g.,

    data_results = ds.data_drift_test(**results.value["MoLGBMRegressor"][0.2]["data_info"])
    data_results.plot("summary")
    data_results.plot(("density", "MedInc"))
    

• table: DataFrame with performance metrics across different data fractions

• options: Dictionary of visualizations configuration for a mulit-line plot where x-axis is the worst fractions (0 to 1), and y-axis is performance metric. Run results.plot() to show this plot.

Examples

TestSuite.compare_residual_cluster(dataset='test', response_type='abs_residual', metric=None, n_clusters=10, cluster_method='ltc', kmedoids_method='pam', sample_size=2000, n_estimators=100, max_depth=5, random_state=0, response_kwargs={}, n_repeats=10, perturb_features=None, perturb_method='normal', noise_level=0.1, alpha=0.1)

Compare model residuals by clustering data points and evaluating performance within clusters.

This test groups samples with similar residual patterns — by learning trajectory (cluster_method="ltc") or feature-space proximity (cluster_method="rf") — and evaluates each model within those clusters, surfacing performance heterogeneity and problematic regions.

Parameters:

• dataset – Dataset to analyze — {"main", "train", "test"}, default "test".

• response_type – Quantity clustered — "abs_residual", "sq_residual", "abs_residual_perturb", "sq_residual_perturb" or "pi_width", default "abs_residual".

• metric – Performance metric (regression default "MSE", classification default "AUC").

• n_clusters – Number of clusters, default 10.

• cluster_method – "ltc" (Learning-Trajectory Cluster) or "rf" (Random-Forest proximity), default "ltc".

• sample_size (kmedoids_method) – KMedoids algorithm ("pam"/"alternate") and proximity sample size — only used when cluster_method="rf".

• max_depth (n_estimators) – Tree count and depth of the underlying ensemble (defaults 100 / 5).

• random_state – Random seed, default 0.

• alpha (response_kwargs n_repeats perturb_features perturb_method noise_level) – Perturbation / prediction-interval controls, used only for the *_perturb and pi_width response types.

Returns:

A ValidationResult with per-cluster metrics for each registered model and the associated comparison visualizations.

TestSuite.compare_fairness(group_config, favorable_label: int = 1, dataset: str = 'test', metric: str = None, threshold: float | int = None)

Compares fairness metrics across multiple models.

This function evaluates and compares fairness metrics for various models based on the provided group configurations, allowing for a comprehensive analysis of model performance across different demographic groups.

Parameters:

group_configdict

Configuration defining protected and reference groups. Each key is a custom group name, and each value is a dictionary with group definitions. Supports three formats:

favorable_label{0, 1}, default=1

For classification: The preferred class label. For regression: 1 means larger predictions are preferred, 0 means smaller predictions are preferred.

dataset{“main”, “train”, “test”}, 1. For numerical features:

{
    "feature": str,           # Feature name
    "protected": {            # Protected group bounds
        "lower": float,       # Lower bound
        "lower_inclusive": bool,
        "upper": float,       # Optional upper bound
        "upper_inclusive": bool
    },
    "reference": {            # Reference group bounds
        "lower": float,       # Optional lower bound
        "lower_inclusive": bool,
        "upper": float,       # Upper bound
        "upper_inclusive": bool
    }
}

2. For categorical features:

   {
       "feature": str,                  # Feature name
       "protected": str or int,         # Protected group category
       "reference": str or int          # Reference group category
   }
   

3. For probabilistic group membership:

   {
       "by_weights": True,
       "protected": str,         # Column name with protected group probabilities
       "reference": str          # Column name with reference group probabilities
   }default="test"
   

The dataset to evaluate fairness on.

metricstr, default=None

Fairness metric to calculate. Higher values indicate less unfairness. If None, defaults are used based on task type.

For regression (default=”SMD”):

• SMD: Standardized Mean Difference (%) between protected and reference groups

For classification (default=”AIR”):

• AIR: Adverse Impact Ratio of predicted probabilities

• PR: Precision Ratio

• RR: Recall Ratio

thresholdfloat or int, default=None

Optional threshold value to display in the visualization. Used to indicate acceptable fairness levels.

Returns:

ValidationResult

A container object with the following components:

• key: “compare_fairness”

• data: Name of the dataset used

• model: List of model names compared

• inputs: Input parameters used

• value: Dictionary of (“<model_name>”, item) pairs, which item is a nested dictionary with dictionary containing the (“<group_name>”, sub_item) pairs for each group; each sub_item contains

  • “fairness_metric”: the fairness metric for each group.

  • “distance”: The KS distance between protected vs reference group predictions.

  • “data_info”: A dictionary containing detailed information about the protected and reference groups, including sample indices and names.

    data_results = ds.data_drift_test(**results.value["MoLGBMClassifier"]["Gender"]["data_info"])
    data_results.plot("summary")
    data_results.plot(("density", "PAY_1"))
    

• table: DataFrame with detailed fairness metrics

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

  • “fairness”: a bar plot where x-axis is the group names, and y-axis is fairness metric

  • “distance”: a bar plot where x-axis is the group names, and y-axis is KS distance metric

Examples