Robustness

Robustness to input perturbations, overall and sliced across data segments, exposed by TestSuite.

TestSuite.diagnose_robustness(dataset: str = 'test', threshold: float = 0.1, metric: str = None, n_repeats: int = 10, perturb_features: str | Tuple = None, perturb_method: str = 'normal', noise_levels: float | int | Tuple = 0.1, random_state: int = 0)

Evaluate model robustness by measuring performance under feature perturbations.

This test assesses how model predictions change when input features are perturbed with different noise levels. It helps identify the model’s stability and sensitivity to input variations.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

Dataset to evaluate robustness on.

metricstr, metric=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

n_repeatsint, default=10

Number of times to repeat the perturbation test for each noise level.

perturb_featuresstr or tuple, default=None

Features to perturb during testing. If None, all features are perturbed. Can be a single feature name or list of feature names.

perturb_method{“normal”, “quantile”}, default=”normal”

Method to perturb numerical features:

• “normal”: Add Gaussian noise scaled by feature standard deviation

• “quantile”: Perturb in quantile space with uniform noise

noise_levelsfloat or tuple, default=0.1

Magnitude of perturbation to apply. Can be a single value or tuple of values.

• For “normal” method: Standard deviation multiplier

• For “quantile” method: Maximum quantile shift

thresholdfloat, default=0.1

Proportion of samples to consider as “Non-robust” cases based on prediction changes. Used for separating samples into “Non-robust” and “Remaining” groups.

random_stateint, default=0

Random seed for reproducible results.

Returns:

ValidationResult

Object containing:

• key: “diagnose_robustness”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the test

• value: Nested dict containing the detailed information about each noise level, e.g., results.value[0.2], which includes items:

  • “score”: The performance metric after perturbing the data with noise level 0.2;

  • “data_info”: The sample indices of small and large prediction changes groups (determined by the threshold), which can be further used for data distribution test, e.g.,

    data_results = ds.data_drift_test(**results.value[0.2]["data_info"])
    data_results.plot("summary")
    data_results.plot(("density", "MedInc"))
    

• table: pd.DataFrame of performance under different noise level and repeat

• options: Dictionary of visualizations configuration for a box plot where x-axis is the noise level, and y-axis is performance metric. Run results.plot() to show this plot.

Examples

TestSuite.diagnose_slicing_robustness(features: str | Tuple = None, dataset: str = 'test', method: str = 'uniform', bins: int | Dict = 10, metric: str = None, n_estimators: int = 1000, threshold: float | int = None, n_repeats: int = 10, perturb_features: str | Tuple = None, perturb_method: str = 'normal', noise_levels: float | int = 0.1, random_state: int = 0)

Get unreliable regions based on one or two slicing features.

This function evaluates the robustness of a model by analyzing its performance across different slices of the dataset defined by the specified features. It computes the metric scores for the slices and identifies regions where the model’s performance is unreliable, allowing for insights into the model’s behavior under various conditions.

Parameters:

featuresUnion[str, Tuple], default=None

Feature names used for slicing. Each tuple element should contain at most 2 features.

• If features=(“X1”, ) or “X1”, computes 1D slicing over X1.

• If features=(“X1”, “X2”), computes 2D slicing over the interaction of X1 and X2.

• If features=((“X1”, ), (“X2”, )), computes 1D slicing over X1 and X2 separately.

Note: Batch mode for 2D slicing is not supported. If None, all 1D features will be used.

dataset{“main”, “train”, “test”}, default=”test”

The data set to be tested.

metricstr, metric=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

method{“uniform”, “quantile”, “auto-xgb1”, “precompute”}, default=”uniform”

Method for binning numerical features:

• “uniform”: Equal-width binning

• “quantile”: Equal-frequency binning (may result in fewer bins due to ties)

• “auto-xgb1”: Use bins of a XGBoost depth-1 model fitted between X and residuals.

• “precompute”: Uses pre-specified bin edges

binsint or dict, default=10

Controls binning granularity:

• If int: Number of bins for numerical features. For “quantile”, this is the maximum number of bins. For “auto-xgb1”, this sets XGBoost’s max_bin parameter.

• If dict: Manual bin specifications for each feature, only used with method=”precompute”. Format: {feature_name: array_of_bin_edges}. Example: {“X0”: [0.1, 0.5, 0.9]} Note: Cannot specify bins for categorical features.

n_estimatorsint, default=1000

The number of estimators in xgboost, used when method=”auto-xgb1”.

thresholdfloat or int, default=None

The metric threshold of non-robust regions. If not specified, it will be the robustness metric of the whole population.

n_repeatsint, default=10

The number of perturbation repetition.

perturb_featuresstr or tuple, default=None

Feature names used for perturbation. If None, all features will be perturbed.

perturb_method{“normal”, “quantile”}, default=”normal”

The perturbation method of numerical features.

noise_levelsfloat or int, default=0.1

The perturbation level.

random_stateint, default=0

The random seed for reproducibility.

Returns:

ValidationResult

A container object with the following components:

• key: “diagnose_slicing_robustness”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the analysis

• value: List of performance metrics for each segment, and each element is a dict containing

  • “Feature”: feature name

  • “Segment”: segment value (categorical) or segment range (numerical)

  • “Size”: number of samples in this segment

  • <”metric”>: perturbed model performance metric value of this segment

  • “Sample_ID”: sample indices of this segment

  • “Sample_Dataset”: dataset name, e.g., “train”, “test”, etc.

  • “Segment_Info”: explicit definition of this segment, similar to “Segment”

  • “Weak”: boolean indicator showing whether this segment is weak or not

• table: DataFrame summarizing slice-wise results

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; To display one preferred plot by results.plot(name=xxx), and the following names are available:

  • None (If only one 1D or 2D slicing features are specified): Performance metric (after perturbation) plot against selected slicing feature(s).

  • “<feature_name>” (If multiple single features are specified): Performance metric (after perturbation) plot against selected slicing feature(s).

Examples