Reliability

Prediction-uncertainty and coverage diagnostics, overall and sliced across data segments, exposed by TestSuite.

TestSuite.diagnose_reliability(train_dataset: str = 'test', test_dataset: str = 'test', test_size: float = 0.5, alpha: float = 0.1, max_depth: int = 5, width_threshold: float = 0.1, random_state: int = 0)

Evaluates model reliability using split conformal prediction.

This method assesses the reliability of model predictions by generating prediction intervals (regression) or prediction sets (classification) using conformal prediction.

Parameters:

train_dataset{“main”, “train”, “test”}, default=”test”

Dataset used for calibrating the conformal prediction model. Not to be confused with the model’s original training set.

test_dataset{“main”, “train”, “test”}, default=”test”

Dataset used for evaluation.

test_sizefloat, default=0.5

Proportion of data to use for testing when train_dataset == test_dataset. Must be between 0 and 1.

alphafloat, default=0.1

Target miscoverage rate (1 - confidence level). For example, alpha=0.1 aims for 90% coverage.

max_depthint, default=5

Maximum depth of the gradient boosting trees for regression tasks. Only used when task_type is REGRESSION.

width_thresholdfloat, default=0.1

For regression: proportion of samples with largest prediction intervals to classify as unreliable.

random_stateint, default=0

Random seed for reproducibility.

Returns:

ValidationResult

A result object containing:

• key: “diagnose_reliability”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the test

• table: DataFrame with average width and coverage metrics

• value: Dictionary containing detailed results including:

  • “interval”: Prediction intervals / sets and related metrics

  • “data_info”: The sample indices of reliable and unreliable samples, which can be further used for data distribution test, e.g.,

    data_results = ds.data_drift_test(**results.value["data_info"])
    data_results.plot("summary")
    data_results.plot(("density", "MedInc"))
    

• options: Dictionary of visualizations configuration for a line plot where x-axis is the actual response, and y-axis is prediction, and prediction interval. Run results.plot() to show this plot.

Notes

For regression tasks:

• Uses residual quantile regression to calculate prediction intervals

• The calibration dataset is split 50/50 into training (for fitting quantile regression model) and validation (for calculating the threshold of nonconformity scores) sets

• Samples with widest prediction intervals are marked as unreliable

For classification tasks:

• Generates prediction sets: {0}, {1}, {0,1}, or {}

• Uses nonconformity scores to determine set membership

• Samples with prediction sets {} or {0,1} are marked as unreliable

Examples

TestSuite.diagnose_slicing_reliability(features: str | Tuple = None, train_dataset: str = 'test', test_dataset: str = 'test', test_size: float = 0.5, method: str = 'uniform', bins: int | Dict = 10, n_estimators: int = 1000, threshold: float | int = None, metric: str = 'width', alpha: float = 0.1, max_depth: int = 5, random_state: int = 0)

Get unreliable regions based on one or two slicing features.

This function analyzes the reliability of a model’s predictions by slicing the dataset based on the provided features. It computes metrics such as width or coverage for the specified bins and identifies regions where the model’s predictions may be unreliable.

Parameters:

featuresUnion[str, Tuple], default=None

Feature names used for slicing. Each tuple element should contain at most 2 features.

• If features=(“X1”, ) or “X1”, computes 1D slicing over X1.

• If features=(“X1”, “X2”), computes 2D slicing over the interaction of X1 and X2.

• If features=((“X1”, ), (“X2”, )), computes 1D slicing over X1 and X2 separately.

Note: Batch mode for 2D slicing is not supported. If None, all 1D features will be used.

train_dataset{“main”, “train”, “test”}, default=”test”

The data set used for training and calibration.

test_dataset{“main”, “train”, “test”}, default=”test”

The data set used for evaluation purpose.

test_sizefloat, default=0.5

Optional test set percentage for splitting the data into train and test sets. Only used when train_dataset == test_dataset.

method{“uniform”, “quantile”, “auto-xgb1”, “precompute”}, default=”uniform”

Method for binning numerical features:

• “uniform”: Equal-width binning

• “quantile”: Equal-frequency binning (may result in fewer bins due to ties)

• “auto-xgb1”: Use bins of a XGBoost depth-1 model fitted between X and residuals.

• “precompute”: Uses pre-specified bin edges

binsint or dict, default=10

Controls binning granularity:

• If int: Number of bins for numerical features. For “quantile”, this is the maximum number of bins. For “auto-xgb1”, this sets XGBoost’s max_bin parameter.

• If dict: Manual bin specifications for each feature, only used with method=”precompute”. Format: {feature_name: array_of_bin_edges}. Example: {“X0”: [0.1, 0.5, 0.9]} Note: Cannot specify bins for categorical features.

n_estimatorsint, default=1000

The number of estimators in xgboost, used when method=”auto-xgb1”.

thresholdfloat or int, default=None

The metric threshold of unreliable regions. If not specified, it will be the reliability metric of the whole population.

metric{“width”, “coverage”}, default=”width”

The metric to be calculated in each slicing bin.

• “width”: The average width in each bin.

• “coverage”: The average coverage in each bin.

alphafloat, default=0.1

The expected coverage of prediction intervals / sets.

max_depthint, default=5

The max_depth parameter of GBM with quantile loss. Only used for regression tasks.

random_stateint, default=0

The random seed for reproducibility.

Returns:

ValidationResult

A container object with the following components:

• key: “diagnose_slicing_reliability”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters used for the analysis

• value: List of performance metrics for each segment, and each element is a dict containing

  • “Feature”: feature name

  • “Segment”: segment value (categorical) or segment range (numerical)

  • “Size”: number of samples in this segment

  • <”metric”>: reliability metric value of this segment

  • “Sample_ID”: sample indices of this segment

  • “Sample_Dataset”: dataset name, e.g., “train”, “test”, etc.

  • “Segment_Info”: explicit definition of this segment, similar to “Segment”

  • “Weak”: boolean indicator showing whether this segment is weak or not

• table: DataFrame with slice analysis results

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; To display one preferred plot by results.plot(name=xxx), and the following names are available:

  • None (If only one 1D or 2D slicing features are specified): Reliability metric plot against selected slicing feature(s).

  • “<feature_name>” (If multiple single features are specified): Reliability metric plot against selected slicing feature(s).

Examples