Test Suite

The TestSuite class (exposed as modeva.TestSuite) provides post-hoc explanation, inherent interpretation, diagnostics and model comparison.

LocalTestSuite

A comprehensive model evaluation and analysis toolkit that provides methods for explaining, diagnosing, comparing, and interpreting machine learning models.

This class serves as the main interface for model evaluation, offering capabilities to: - Explain model behavior using various techniques (PFI, SHAP, LIME, etc.) - Diagnose model performance (accuracy, reliability, robustness, fairness, etc.) - Compare multiple models across different metrics - Interpret model decisions and feature importance - Register and manage test results using MLflow

Parameters

dataset : LocalDataSet, optional
The dataset object to be used for evaluation

model : ModelBase, optional
The primary model to be evaluated

models : List[ModelBase], optional
List of models for comparison purposes

name : str, default=“testsuite”
Name of the testsuite, used for MLflow experiment tracking

Examples

>>> # Create a testsuite for single model evaluation >>> testsuite = LocalTestSuite(dataset=my_dataset, model=my_model, name=“model_evaluation”) >>> >>> # Run accuracy diagnosis >>> result = testsuite.diagnose_accuracy_table() >>> >>> # Compare multiple models >>> comparison_sheet = LocalTestSuite(dataset=my_dataset, models=[model1, model2]) >>> comparison = comparison_sheet.compare_accuracy_table()

__init__(dataset: local_dataset.LocalDataSet=None, model: base.ModelBase=None, models: List[base.ModelBase]=None, name: str='testsuite')

get_dataset()

Return the dataset object.

set_dataset(dataset: local_dataset.LocalDataSet)

Set dataset for test suite.

Parameters

dataset : Dataset
Dataset object

get_model()

Return the model object.

set_model(model: base.ModelBase)

Set model for test suite.

Parameters

model : ModelBase
modeva built-in models or external models that implement ModelBase

explain_pfi(dataset: str='test', sample_size: int=5000, n_repeats: int=10, random_state: int=0)

explain_hstatistic(features: Union[Tuple, List]=None, dataset: str='test', sample_size: int=5000, percentiles: Tuple=(0, 1), grid_resolution: int=10, response_method: str='auto', random_state: int=0)

explain_pdp(features: Union[str, Tuple[str]]=None, dataset: str='test', sample_size: int=5000, percentiles: Tuple=(0, 1), grid_resolution: int=20, response_method: str='auto', random_state: int=0)

explain_ale(features: Union[str, Tuple[str]]=None, dataset: str='test', sample_size: int=5000, grid_resolution: int=20, response_method: str='auto', random_state: int=0)

explain_lime(dataset: str='test', sample_index: int=0, centered: bool=True, random_state: int=0)

explain_shap(dataset: str='test', sample_index: int=0, baseline_dataset: str='train', baseline_sample_index: int=None, baseline_sample_size: int=500, random_state: int=0)

diagnose_accuracy_table(train_dataset: str='train', test_dataset: str='test', metric: Union[str, Tuple]=None)

diagnose_residual_analysis(features: str=None, use_prediction: bool=False, dataset: str='test', sample_size: int=2000, random_state: int=0)

diagnose_residual_interpret(dataset: str='test', n_estimators: int=100, max_depth: int=2, **xgb_kwargs)

diagnose_residual_cluster(dataset: str='test', response_type: str='abs_residual', metric: str=None, n_clusters: int=10, cluster_method: str='ltc', kmedoids_method: str='pam', sample_size: int=2000, n_estimators: int=100, max_depth: int=5, random_state: int=0, n_repeats: int=10, perturb_features: Union[str, Tuple]=None, perturb_method: str='normal', noise_level: Union[float, int]=0.1, alpha: float=0.1)

diagnose_weakness_region(train_dataset='train', test_dataset='test', metric=None, geometry_method='arf', bins=10, weak_fraction=0.2, top_n_features=10, min_count=20, geometry_n_estimators=120, geometry_max_depth=12, geometry_min_samples_leaf=20, geometry_num_trees=30, geometry_max_iters=10, mi_n_estimators=200, mi_max_depth=0, mi_min_samples_leaf=10, mi_n_splits=5, random_state=0)

diagnose_slicing_accuracy(features: Union[str, Tuple]=None, dataset: str='test', metric: str=None, method: str='uniform', bins: Union[int, Dict]=10, n_estimators: int=1000, threshold: Union[float, int]=None)

diagnose_slicing_overfit(features: Union[str, Tuple]=None, train_dataset: str='train', test_dataset: str='test', metric: str=None, method: str='uniform', bins: Union[int, Dict]=10, n_estimators: int=1000, threshold: Union[float, int]=None)

diagnose_slicing_reliability(features: Union[str, Tuple]=None, train_dataset: str='test', test_dataset: str='test', test_size: float=0.5, method: str='uniform', bins: Union[int, Dict]=10, n_estimators: int=1000, threshold: Union[float, int]=None, metric: str='width', alpha: float=0.1, max_depth: int=5, random_state: int=0)

diagnose_slicing_robustness(features: Union[str, Tuple]=None, dataset: str='test', method: str='uniform', bins: Union[int, Dict]=10, metric: str=None, n_estimators: int=1000, threshold: Union[float, int]=None, n_repeats: int=10, perturb_features: Union[str, Tuple]=None, perturb_method: str='normal', noise_levels: Union[float, int]=0.1, random_state: int=0)

diagnose_slicing_fairness(group_config, features: Union[str, Tuple]=None, favorable_label: int=1, dataset: str='test', metric: str=None, method: str='uniform', bins: Union[int, Dict]=10, n_estimators: int=1000, threshold: Union[float, int]=None)

diagnose_robustness(dataset: str='test', threshold: float=0.1, metric: str=None, n_repeats: int=10, perturb_features: Union[str, Tuple]=None, perturb_method: str='normal', noise_levels: Union[float, int, Tuple]=0.1, random_state: int=0)

diagnose_reliability(train_dataset: str='test', test_dataset: str='test', test_size: float=0.5, alpha: float=0.1, max_depth: int=5, width_threshold: float=0.1, random_state: int=0)

diagnose_resilience(dataset: str='test', method: str='worst-sample', metric: str=None, alphas: tuple=None, n_clusters: int=10, random_state: int=0)

diagnose_fairness(group_config, favorable_label: int=1, dataset: str='test', metric: str=None, threshold: Union[float, int]=None)

diagnose_mitigate_unfair_thresholding(group_config, favorable_label: int=1, dataset: str='test', metric: str=None, performance_metric: str=None, proba_cutoff: Union[int, Tuple]=None)

diagnose_mitigate_unfair_binning(group_config, favorable_label: int=1, dataset: str='test', metric: str=None, performance_metric: str=None, binning_features: Union[str, Tuple]=None, binning_method: str='uniform', bins: Union[int, Dict]=10)

compare_accuracy_table(train_dataset: str='train', test_dataset: str='test', metric: Union[str, Tuple]=None)

compare_slicing_accuracy(features: str, dataset: str='test', metric: str=None, method: str='uniform', bins: Union[int, Dict]=10, n_estimators: int=1000, threshold: Union[float, int]=None)

compare_slicing_overfit(features: str, train_dataset: str='train', test_dataset: str='test', metric: str=None, method: str='uniform', bins: Union[int, Dict]=10, n_estimators: int=1000, threshold: Union[float, int]=None)

compare_slicing_reliability(features: str, train_dataset: str='test', test_dataset: str='test', test_size: float=0.5, method: str='uniform', bins: Union[int, Dict]=10, n_estimators: int=1000, threshold: Union[float, int]=None, metric: str='width', alpha: float=0.1, max_depth: int=5, random_state: int=0)

compare_slicing_robustness(features: str, dataset: str='test', metric: str=None, method: str='uniform', bins: Union[int, Dict]=10, n_estimators: int=1000, threshold: Union[float, int]=None, n_repeats: int=10, perturb_features: Union[str, Tuple]=None, perturb_method: str='normal', noise_levels: Union[float, int]=0.1, random_state: int=0)

compare_slicing_fairness(group_config, features: str, favorable_label: int=1, dataset: str='test', metric: str=None, method: str='uniform', bins: Union[int, Dict]=10, n_estimators: int=1000, threshold: Union[float, int]=None)

compare_robustness(dataset: str='test', metric: str=None, n_repeats: int=10, perturb_features: Union[str, Tuple]=None, perturb_method: str='normal', noise_levels: Union[float, int, Tuple]=0.1, random_state: int=0)

compare_reliability(train_dataset: str='test', test_dataset: str='test', test_size: float=0.5, alpha: float=0.1, max_depth: int=5, random_state: int=0)

compare_resilience(dataset: str='test', method: str='worst-sample', metric: str=None, alphas: tuple=None, n_clusters: int=10, random_state: int=0)

compare_residual_cluster(dataset: str='test', response_type: str='abs_residual', metric: str=None, n_clusters: int=10, cluster_method: str='ltc', kmedoids_method: str='pam', sample_size: int=2000, n_estimators: int=100, max_depth: int=5, random_state: int=0, n_repeats: int=10, perturb_features: Union[str, Tuple]=None, perturb_method: str='normal', noise_level: Union[float, int]=0.1, alpha: float=0.1)

compare_fairness(group_config, favorable_label: int=1, dataset: str='test', metric: str=None, threshold: Union[float, int]=None)

get_main_effects()

get_interactions()

interpret_effects(features: Union[str, Tuple]=None, dataset: str='test', grid_size: int=200)

interpret_ei(dataset: str='test')

interpret_local_ei(dataset: str='test', sample_index: int=0)

interpret_coef(features: Union[str, Tuple[str]]=None)

interpret_fi(dataset: str='test')

interpret_local_fi(dataset: str='test', sample_index: int=0, centered: bool=True)

interpret_local_linear_fi(dataset: str='test', sample_index: int=0, centered: bool=True)

interpret_llm_summary(dataset: str='test')

interpret_llm_pc(dataset: str='test')

interpret_llm_profile(feature: str=None, dataset: str='test')

interpret_llm_violin(feature: str=None, dataset: str='test')

interpret_global_tree()

interpret_local_tree(dataset: str='test', sample_index: int=0)

interpret_local_moe_weights(dataset: str='test', sample_index: int=0)

interpret_effects_moe_average(features: Union[str, Tuple], dataset: str='test', grid_size: int=100, sample_size: int=5000, random_state: int=0)

interpret_moe_cluster_analysis(dataset: str='test', metric: str=None)

list(cls)

List all the experiments saved in database.

Returns

pd.DataFrame
A table showing the experiments details in database.

register(name: str, test_result: ValidationResult, register_name: str=None, description: str=None, tags: Optional[Dict[str, Any]]=None, run_id: str=None)

Register a test into MLFlow.

Parameters

name : str
The current name of the test to be registered.

test_result : ValidationResult
The validation result object of test.

register_name : str, default=None
The register name of the test in MLFlow. If None, will be the same as name.

description : str, default=None
The description of this test.

tags : dict, default=None
The tags.

run_id : str, default=None
The run id in MLFLow.

delete_registed_test(name, run_id: str=None)

Load config and result of registered tests.

Parameters

name : str
The name of test used for filtering.

run_id : str, default=None
run id of the registered test.

load_registered_test(name: str, run_id: str=None)

Load config and result of registered tests.

Parameters

name : str
The name of test used for filtering.

run_id : str, default=None
run id of the registered test.

list_registered_tests(name: str=None)

Return the list all registered tests.

Parameters

name : str
The name of test used for filtering.

display_test_results(testsuite_name, test_list: list=None)

Get ValidationResult object of registered test.

Parameters

testsuite_name : str
The testsuite to display.

test_list : list
The content list of test, e.g.,

[{'name': 'test1', 'run_id': 'xxx', 'display_table': True, 'display_plot': True},
 {'name': 'test2', 'display_table': True, 'display_plot': False}]

or

['test1', 'test2']

export_report(path: str='report.html')

Export report to html

Parameters

path : str, optional
The export path, by default “report.html”