Weakness Region Analysis (Classification)

This example demonstrates AMIF (Adversarial Mutual Information Forest) weakness region diagnostics for classification models. In addition to the regression diagnostics, classification includes confusion matrix comparisons between the full test set and weak regions.

We use the TaiwanCredit dataset with an XGBoost classifier.

Setup

Import libraries and suppress warnings.

import warnings
warnings.filterwarnings("ignore")

from modeva import DataSet, TestSuite
from modeva.models import MoXGBClassifier

Load Dataset

Load the TaiwanCredit dataset and set up for classification.

ds = DataSet()
ds.load("TaiwanCredit")
ds.set_target(feature="FlagDefault")
ds.set_task_type("Classification")
ds.set_random_split(test_ratio=0.2)

print(f"Train: {ds.train_x.shape}, Test: {ds.test_x.shape}")

Train: (24000, 23), Test: (6000, 23)

Train Model

Train an XGBoost classifier.

model = MoXGBClassifier(
    name="XGBoost", n_estimators=200, max_depth=4, learning_rate=0.1
)
model.fit(ds.train_x, ds.train_y.ravel())

MoXGBClassifier(base_score=None, booster=None, callbacks=None,
                colsample_bylevel=None, colsample_bynode=None,
                colsample_bytree=None, device=None, early_stopping_rounds=None,
                enable_categorical=False, eval_metric=None, feature_types=None,
                gamma=None, grow_policy=None, importance_type=None,
                interaction_constraints=None, learning_rate=0.1, max_bin=None,
                max_cat_threshold=None, max_cat_to_onehot=None,
                max_delta_step=None, max_depth=4, max_leaves=None,
                min_child_weight=None, missing=nan, monotone_constraints=None,
                multi_strategy=None, n_estimators=200, n_jobs=None,
                num_parallel_tree=None, objective='binary:logistic', ...)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Run Weakness Region Diagnostics

Use diagnose_weakness_region with the CARF geometry method and ACC metric. The min_count parameter sets the minimum number of samples required per region bin to compute a valid metric.

ts = TestSuite(ds, model)

result = ts.diagnose_weakness_region(
    geometry_method="carf",
    metric="ACC",
    bins=8,
    weak_fraction=0.2,
    top_n_features=10,
    min_count=15,
)

Result Table

The result table summarizes performance across all 2D grid regions.

result.table

	Geometry Bin	MI Bin	ACC (Train)	ACC (Test)	Sample Count	Is Weak
0	1	0	NaN	NaN	22	False
1	6	7	0.9167	NaN	45	False
2	7	7	0.8333	NaN	19	False
3	1	1	0.7457	0.6182	346	True
4	7	0	0.8047	0.6423	722	True
5	4	0	0.6805	0.6495	604	True
6	5	0	0.7380	0.6538	562	True
7	2	1	0.6818	0.6555	1001	True
8	3	0	0.7186	0.6579	915	True
9	6	0	0.7342	0.6583	515	True
10	6	1	0.7974	0.6709	464	True
11	2	0	0.6707	0.6709	410	True
12	3	1	0.7328	0.6970	477	True
13	1	2	0.7570	0.7093	407	True
14	5	1	0.7482	0.7155	533	True
15	2	2	0.7130	0.7315	703	False
16	1	3	0.7759	0.7531	380	False
17	4	1	0.7557	0.7558	483	False
18	7	1	0.7838	0.8026	446	False
19	7	2	0.8578	0.8058	518	False
20	2	3	0.8443	0.8060	279	False
21	1	4	0.8485	0.8152	389	False
22	4	7	0.8667	0.8261	128	False
23	0	5	0.8909	0.8291	603	False
24	5	3	0.8807	0.8304	615	False
25	5	2	0.8089	0.8346	630	False
26	6	2	0.8486	0.8467	553	False
27	3	6	0.8898	0.8472	426	False
28	5	6	0.8992	0.8491	301	False
29	4	6	0.8739	0.8500	449	False
30	5	4	0.8172	0.8532	563	False
31	4	2	0.7500	0.8571	522	False
32	3	3	0.8514	0.8600	470	False
33	4	3	0.8679	0.8632	534	False
34	1	7	0.8691	0.8641	879	False
35	6	3	0.8722	0.8699	710	False
36	2	6	0.9018	0.8824	438	False
37	4	5	0.8612	0.8842	484	False
38	4	4	0.8756	0.8854	546	False
39	0	4	0.9315	0.8889	91	False
40	3	2	0.8174	0.8889	417	False
41	5	5	0.9013	0.8913	477	False
42	7	5	0.9208	0.8989	455	False
43	3	7	0.8702	0.9000	161	False
44	3	4	0.8943	0.9000	507	False
45	7	3	0.9028	0.9034	762	False
46	1	5	0.9174	0.9048	577	False
47	2	4	0.9033	0.9103	347	False
48	6	4	0.8881	0.9124	673	False
49	7	4	0.9051	0.9141	634	False
50	3	5	0.9023	0.9143	377	False
51	2	5	0.8871	0.9167	320	False
52	1	6	0.8894	0.9185	750	False
53	0	7	0.8848	0.9197	2197	False
54	0	6	0.9098	0.9290	859	False
55	6	5	0.9103	0.9326	457	False
56	2	7	0.8646	0.9333	252	False
57	5	7	0.9074	0.9333	69	False
58	6	6	0.8864	0.9420	333	False
59	7	6	0.9189	0.9565	194	False

Print the weak region summary statistics.

v = result.value
print(f"Weak test samples: {v['n_weak_samples']} / {v['n_total_samples']} "
      f"({100*v['n_weak_samples']/v['n_total_samples']:.1f}%)")
print(f"Metric: {v['metric']}, Cutoff: {v['cutoff']:.4f}")

Weak test samples: 1357 / 6000 (22.6%)
Metric: ACC, Cutoff: 0.7187

Region Performance Heatmaps

Heatmaps show model performance across the 2D geometry-MI grid.

result.plot(name="region_performance_train", figsize=(6.5, 5))

Test performance heatmap.

result.plot(name="region_performance_test", figsize=(6.5, 5))

Sample count per region.

result.plot(name="region_sample_count", figsize=(6.5, 5))

Feature Rankings

JS divergence ranking shows which features differ most between weak and non-weak regions.

result.plot(name="js_divergence_ranking", figsize=(6.5, 4))

MI importance ranking.

result.plot(name="mi_importance_ranking", figsize=(6.5, 4))

Score Distributions

Distribution of geometry and MI scores across the dataset.

result.plot(name="geometry_score_distribution", figsize=(6.5, 4))

MI score distribution.

result.plot(name="mi_score_distribution", figsize=(6.5, 4))

Confusion Matrices

Compare confusion matrices between the full test set and the weak region. This reveals whether certain classes are disproportionately affected.

result.plot(name=("confusion_matrix", "all_test"), figsize=(5, 4))

Confusion matrix for weak region samples only.

result.plot(name=("confusion_matrix", "weak_test"), figsize=(5, 4))

Feature Distributions (Weak vs. Rest)

Compare feature distributions between weak and non-weak samples for the top 3 most divergent features.

feat_figs = [
    f for f in result.get_figure_names()
    if isinstance(f, tuple) and f[0] == "feature_distribution"
]
for fig_name in feat_figs[:3]:
    result.plot(name=fig_name, figsize=(6.5, 4))