Overfit

Slicing-based overfit diagnostics that compare train and test performance across data segments, exposed by TestSuite.

TestSuite.diagnose_slicing_overfit(features: str | Tuple = None, train_dataset: str = 'train', test_dataset: str = 'test', metric: str = None, method: str = 'uniform', bins: int | Dict = 10, n_estimators: int = 1000, threshold: float | int = None)

Identify overfit regions based on one or two slicing features.

This method analyzes the performance gap between training and testing datasets for specified features, helping to identify potential overfitting.

Parameters:
featuresUnion[str, Tuple], default=None

Feature names used for slicing. Each tuple element should contain at most 2 features.

  • If features=(“X1”, ) or “X1”, computes 1D slicing over X1.

  • If features=(“X1”, “X2”), computes 2D slicing over the interaction of X1 and X2.

  • If features=((“X1”, ), (“X2”, )), computes 1D slicing over X1 and X2 separately.

Note: Batch mode for 2D slicing is not supported. If None, all 1D features will be used.

train_datasetstr, default=”train”

The dataset used for training. Options include “main”, “train”, or “test”.

test_datasetstr, default=”test”

The dataset used for testing. Options include “main”, “train”, or “test”.

metricstr, metric=None

Model performance metric to use.

  • For classification (default=”AUC”): “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”

  • For regression (default=”MSE”): “MSE”, “MAE”, and “R2”

methodstr, default=”uniform”

The binning method for numerical features. Options include:

  • “uniform”: Equal-width bins

  • “quantile”: Equal-frequency bins

  • “auto-xgb1”: Binning method for XGBoost

  • “precompute”: Predefined bins

binsint or dict, default=10

Controls binning granularity:

  • If int: Number of bins for numerical features. For “quantile”, this is the maximum number of bins. For “auto-xgb1”, this sets XGBoost’s max_bin parameter.

  • If dict: Manual bin specifications for each feature, only used with method=”precompute”. Format: {feature_name: array_of_bin_edges}. Example: {“X0”: [0.1, 0.5, 0.9]} Note: Cannot specify bins for categorical features.

n_estimatorsint, default=1000

The number of estimators to use in XGBoost when method=”auto-xgb1”.

thresholdfloat or int, default=None

The metric gap threshold for identifying weak regions. If not specified, it will be the performance metric gap of the whole population.

Returns:
ValidationResult

An object containing the results of the slicing overfit detection, including:

  • key: “diagnose_slicing_overfit”

  • data: Name of the dataset used

  • model: Name of the model used

  • inputs: Input parameters used for the test

  • value: List of performance metrics for each segment, and each element is a dict containing

    • “Feature”: feature name

    • “Segment”: segment value (categorical) or segment range (numerical)

    • “Size”: number of samples in this segment

    • <”metric”>: performance metric gap value of this segment

    • “Sample_ID”: sample indices of this segment

    • “Sample_Dataset”: dataset name, e.g., “train”, “test”, etc.

    • “Segment_Info”: explicit definition of this segment, similar to “Segment”

    • “Weak”: boolean indicator showing whether this segment is weak or not

  • table: pd.DataFrame summarizing the performance metrics for both training and testing datasets, including the calculated gaps.

  • options: Dictionary of visualizations configuration. Run results.plot() to show all plots; To display one preferred plot by results.plot(name=xxx), and the following names are available:

    • None (If only one 1D or 2D slicing features are specified): Performance gap plot against selected slicing feature(s).

    • “<feature_name>” (If multiple single features are specified): Performance gap plot against selected slicing feature(s).

Examples