Weakness Detection

TestSuite.diagnose_weakness_region(train_dataset='train', test_dataset='test', metric=None, geometry_method='arf', bins=10, weak_fraction=0.2, top_n_features=10, min_count=20, geometry_n_estimators=120, geometry_max_depth=12, geometry_min_samples_leaf=20, geometry_num_trees=30, geometry_max_iters=10, mi_n_estimators=200, mi_max_depth=0, mi_min_samples_leaf=10, mi_n_splits=5, random_state=0)

Diagnose model weakness regions using AMIF (Adversarial Mutual Information Forest).

AMIF partitions the data into a 2D grid (geometry score × MI score) and identifies regions where the model performs poorly. For each grid cell, the model metric is computed on train and test subsets; weak regions are those in the bottom fraction of test performance. Features are then ranked by Jensen–Shannon divergence between the weak-region distribution and the rest, highlighting which features differ most where the model struggles.

Parameters:

train_dataset – Dataset split used for training performance — {"main", "train", "test"}, default "train".
test_dataset – Dataset split used for test performance — {"main", "train", "test"}, default "test".
metric – Metric name; defaults to "MSE" for regression and "ACC" for classification.
geometry_method – Geometry scoring method — "arf" (Adversarial Random Forest) or "carf" (Custom Adversarial Random Forest), default "arf".
bins – Number of quantile bins for each axis of the 2D grid, default 10.
weak_fraction – Fraction of test bins (by metric) considered weak, default 0.2.
top_n_features – Number of top features to show in the JS-divergence ranking, default 10.
min_count – Minimum sample count per bin to compute the metric, default 20.
geometry_min_samples_leaf (geometry_n_estimators geometry_max_depth) – CARF geometry-scorer settings (defaults 120 / 12 / 20).
geometry_max_iters (geometry_num_trees) – ARF geometry-scorer settings (defaults 30 / 10).
mi_n_splits (mi_n_estimators mi_max_depth mi_min_samples_leaf) – MI-scorer settings (defaults 200 / 0 / 10 / 5); mi_max_depth=0 means unlimited.
random_state – Random seed for reproducibility, default 0.

Returns:

A ValidationResult whose table holds per-bin train/test metrics and a Is Weak flag, and whose value/options carry the geometry and MI scores, weak-bin masks, JS and MI feature rankings, and the associated mocharts visualizations.