Residual Analysis

Residual diagnostics, residual interpretation and residual clustering, exposed by TestSuite.

TestSuite.diagnose_residual_analysis(features: str = None, use_prediction: bool = False, dataset: str = 'test', sample_size: int = 2000, random_state: int = 0)

Analyze the relationship between model residuals and a specified feature.

Creates a scatter plot showing the residuals (actual - predicted values) against a chosen feature or target variable. This can help identify patterns or heteroscedasticity in the model’s predictions. For classification tasks, residuals are calculated using predicted probabilities of the positive class.

Parameters:

featuresstr, default=None

The name of the feature to plot on the x-axis. Can be ignored when use_prediction is True.

use_prediction: bool, default=False

Whether to use the model prediction (predicted probability for classification) as x-axis.

dataset{“main”, “train”, “test”}, default=”test”

Which dataset partitionto use for the analysis.

sample_sizeint, default=2000

Maximum number of points to plot. If the dataset is larger, a random subsample of this size will be used to improve visualization clarity.

random_stateint, default=0

Random seed for reproducible subsampling.

Returns:

ValidationResult

A container object with the following attributes:

• key: “diagnose_residual”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the test

• value : Dictionary containing the x-axis values and residuals used in the plot

• table : DataFrame containing the plotted data

• options: Dictionary of visualizations configuration for a scatter plot where x-axis is the selected feature value, and y-axis is prediction residual (y - y_hat). Run results.plot() to show this plot.

Notes

For classification models, residuals are calculated as the difference between the actual class labels and the predicted probabilities for the positive class. For regression models, residuals are the difference between actual and predicted values.

Examples

TestSuite.diagnose_residual_interpret(dataset: str = 'test', n_estimators: int = 100, max_depth: int = 2, **xgb_kwargs)

Analyzes feature importance by examining their relationship with prediction residuals.

This method calculates how much each feature contributes to explaining the model’s prediction errors (residuals). A higher importance score indicates the feature has a stronger relationship with prediction errors.

As method is one of {“uniform”, “quantile”, “precompute”}, this test performs binning to each predictor variable, and then transform the binning results using one-hot encoding. The encoded varialbes are then fitted to the residual using l2-regularized linear model. The importance of each predictor (under the framework of functional ANOVA) are aggregated using the linear coefficients.

As the method is “auto-xgb1”, then a xgboost depth-1 model is used to fit predictors and the residual. And the feature importance of the xgboost model (under the framework of functional ANOVA) is used as final feature importance.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

Which dataset to analyze.

n_estimatorsint, default=100

Number of trees in Xgboost.

max_depthint, default=2

The maximum tree depth in Xgboost.

**xgb_kwargs:

Other hyperparameters for xgboost.

Returns:

ValidationResult

Contains:

• key: “diagnose_residual_interpret”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the test

• value: Dictionary containing:

  “Model”: the fitted xgboost model object “DataSet”: the dataset that contains residual as response “Feature Importance”: A dict with list of feature names and residual feature importance “Effect Importance”: A dict with list of effect names and residual effect importance

• options: Dictionary of visualizations configuration. Run results.plot(name=xxx) to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

  • “feature_importance”: feature importance plot.

  • “effect_importance”: effect importance plot.

Notes

The feature importance is calculated as the normalized variance of predictions when using each feature alone. For methods other than “auto-xgb1”, features are first binned then one-hot encoded before fitting a Ridge regression model to predict absolute residuals.

Examples

TestSuite.diagnose_residual_cluster(dataset: str = 'test', response_type: str = 'abs_residual', metric: str = None, n_clusters: int = 10, cluster_method: str = 'ltc', kmedoids_method: str = 'pam', sample_size: int = 2000, n_estimators: int = 100, max_depth: int = 5, random_state: int = 0, n_repeats: int = 10, perturb_features: str | Tuple = None, perturb_method: str = 'normal', noise_level: float | int = 0.1, alpha: float = 0.1)

Analyze model residuals by clustering data points and evaluating performance within clusters.

This test identifies groups of samples with similar residual patterns by clustering data points based on their learning trajectories or proximity in feature space. It helps diagnose model performance heterogeneity across different data regions and identify problematic clusters where the model performs poorly.

Parameters:

dataset{“main”, “train”, “test”}, default=”test”

Dataset to analyze.

response_type: str, default=”abs_residual”

The response type, options include

• “abs_residual”: absolute residual

• “sq_residual”: squared residual

• “abs_residual_perturb”: absolute residual after X perturbation as used in robustness test

• “sq_residual_perturb”: squared residual after X perturbation

• “pi_width”: prediction interval width obtained as used in reliability test; note that as dataset=”test”, the test data will be split for calibration (conformal prediction), so that the calibration set is excluded in the final reported results.

metricstr, metric=None

Model performance metric to use.

• For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

• For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

n_clustersint, default=10

Number of clusters to create.

cluster_method{‘ltc’, ‘rf’}, default=’ltc’

Which algorithm to use.

• ‘ltc’: This method (Learning Trajectory Cluster; LTC) fits a gradient boosting models between predictors and the response_type. It extracts prediction trajectories during training, applies optional weighting schemes, performs PCA dimensionality reduction, and clusters samples based on their learning patterns.

• ‘rf’: This method (Random Forest; RF) fits a Random Forest between predictors and the response_type. Then, it generates a proximity matrix based on fitted trees, and clusters the distance matrix using KMedoids.

kmedoids_method{‘alternate’, ‘pam’}, default=’pam’

Which algorithm to use in KMedoids. ‘alternate’ is faster while ‘pam’ is more accurate. Only used when cluster_method=’rf’.

sample_sizeint, default=2000

sample size for speedup the calculation of the proximity matrix and clustering. Only used when cluster_method=’rf’.

n_estimatorsint, default=100

Number of trees in the Random Forest (cluster_method=’rf’) or gradient boosting models (cluster_method=’ltc’).

max_depthint, default=5

Maximum depth of trees in the Random Forest (cluster_method=’rf’) or gradient boosting models (cluster_method=’ltc’).

random_stateint, default=0

Random seed for reproducibility.

response_kwargsdict, default={}

Addition arguments for calculating the response.

n_repeatsint, default=10

Number of times to repeat the perturbation test for each noise level. Only used as response_type=”abs_residual_perturb” or “sq_residual_perturb”.

perturb_featuresstr or tuple, default=None

Features to perturb during testing. If None, all features are perturbed. Can be a single feature name or list of feature names. Only used as response_type=”abs_residual_perturb” or “sq_residual_perturb”.

perturb_method{“normal”, “quantile”}, default=”normal”

Method to perturb numerical features:

• “normal”: Add Gaussian noise scaled by feature standard deviation

• “quantile”: Perturb in quantile space with uniform noise

Only used as response_type=”abs_residual_perturb” or “sq_residual_perturb”.

noise_levelfloat, default=0.1

Magnitude of perturbation to apply. Can be a single value or tuple of values.

• For “normal” method: Standard deviation multiplier

• For “quantile” method: Maximum quantile shift

Only used as response_type=”abs_residual_perturb” or “sq_residual_perturb”.

alphafloat, default=0.1

Target miscoverage rate (1 - confidence level). For example, alpha=0.1 aims for 90% coverage. Only used as response_type=”pi_width”.

Returns:

ValidationResult

Contains:

• key: “diagnose_resilience_cluster”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the test

• value: Dict with clustering results including feature importance, embeddings, and per-cluster performance

  • “feature_importance”: residual feature importance by random forest

  • “cluster_X”: X being clustered

  • “cluster_y”: y being clustered

  • “cluster_sample_weight”: sample_weight being clustered

  • “cluster_labels”: cluster labels of each sample

  • “cluster_pred_func”: the function that receives X as input and output the cluster ID

  • “clusters”: Nested dict containing the detailed information about each cluster, the i-th cluster can be accessed via its cluster id, i.e., results.value[“cluster”][i], which includes items:

    • “score”: The performance metric of this cluster;

    • “data_info”: The sample indices within and outside this cluster, which can be further used for data distribution test, e.g.,

    data_results = ds.data_drift_test(**results.value["clusters"][2]["data_info"])
    data_results.plot("summary")
    data_results.plot(("density", "MedInc"))
    

• table: DataFrame with performance metrics for each cluster

• options: Dictionary of visualizations configuration. Run results.plot(name=xxx) to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

  • “cluster_residual”: Bar plot of residual for each cluster.

  • “cluster_performance”: Bar plot visualizing the performance scores against each cluster.

  • “feature_importance”: feature importance plot.

Notes

When response_type = “pi_width” and dataset=”test”, the test data will be split for calibration (conformal prediction), so that the calibration set is excluded in the final reported results.

Examples