AMIF Weakness Region Diagnostics

Overview

AMIF (Adversarial Mutual Information Forest) identifies model weakness regions by partitioning data into a 2D grid based on two independent scoring axes:

Geometry score — data density estimated via Adversarial Random Forest (ARF) or Conditional ARF (CARF)
MI score — predictive mutual information estimated via cross-validated Random Forest

Regions where the model performs worst are flagged as “weak,” enabling targeted diagnosis of where and why a model fails.

Method

The AMIF pipeline consists of four steps:

Geometry Scoring: An ARF or CARF model estimates each sample’s density score. Low-density regions correspond to sparse or out-of-distribution data.
MI Scoring: A Random Forest cross-validates predictions to estimate each sample’s predictive information. Low MI indicates features carry little information about the target in that region.
2D Binning: Samples are placed into a grid of bins x bins regions based on their geometry and MI scores. Per-region metrics are computed.
Weak Region Identification: Regions with the worst weak_fraction of performance are flagged. Feature distributions are compared between weak and non-weak samples using Jensen-Shannon divergence.

Implementation in MoDeVa

from modeva import DataSet, TestSuite
from modeva.models import MoXGBRegressor

# Load data
ds = DataSet()
ds.load("CaliforniaHousing")
ds.set_target(feature="MedHouseVal")
ds.set_task_type("Regression")
ds.set_random_split(test_ratio=0.2)

# Train model
model = MoXGBRegressor(
    name="XGBoost", n_estimators=200, max_depth=5, learning_rate=0.1
)
model.fit(ds.train_x, ds.train_y.ravel())

# Run weakness region diagnostics
ts = TestSuite(ds, model)
result = ts.diagnose_weakness_region(
    geometry_method="carf",
    metric="MSE",
    bins=8,
    weak_fraction=0.2,
    top_n_features=8,
)

# View result summary
result.table

Output Interpretation

The diagnose_weakness_region method returns a ValidationResult with multiple plot types:

Region Performance Heatmaps

Three heatmaps over the 2D geometry-MI grid:

region_performance_train — training metric per region
region_performance_test — test metric per region
region_sample_count — number of samples per region

Comparing train vs test heatmaps reveals overfitting patterns: regions with good training performance but poor test performance indicate overfitting.

result.plot(name="region_performance_train", figsize=(6.5, 5))
result.plot(name="region_performance_test", figsize=(6.5, 5))
result.plot(name="region_sample_count", figsize=(6.5, 5))

JS Divergence Ranking

Ranks features by Jensen-Shannon divergence between their distributions in weak vs non-weak regions. High JS divergence indicates the feature has a very different distribution in weak regions, making it a key driver of model weakness.

result.plot(name="js_divergence_ranking", figsize=(6.5, 4))

MI Importance Ranking

Ranks features by their mutual information with the target variable. Features with high MI are important predictors overall.

result.plot(name="mi_importance_ranking", figsize=(6.5, 4))

Score Distributions

Distribution plots for both scoring axes, showing how geometry and MI scores are spread across the dataset.

result.plot(name="geometry_score_distribution", figsize=(6.5, 4))
result.plot(name="mi_score_distribution", figsize=(6.5, 4))

Feature Distributions

Side-by-side comparison of feature distributions between weak and non-weak samples. Available for the top N features ranked by JS divergence.

feat_figs = [
    f for f in result.get_figure_names()
    if isinstance(f, tuple) and f[0] == "feature_distribution"
]
for fig_name in feat_figs[:3]:
    result.plot(name=fig_name, figsize=(6.5, 4))

Confusion Matrices (Classification Only)

For classification tasks, confusion matrices are available for the full test set and the weak region subset:

result.plot(name=("confusion_matrix", "all_test"), figsize=(5, 4))
result.plot(name=("confusion_matrix", "weak_test"), figsize=(5, 4))

Parameters

The diagnose_weakness_region method accepts the following parameters:

Parameter	Default	Description
`train_dataset`	`"train"`	Dataset split for training performance (`"main"`, `"train"`, or `"test"`)
`test_dataset`	`"test"`	Dataset split for test performance (`"main"`, `"train"`, or `"test"`)
`metric`	Task-dependent	Evaluation metric (`"MSE"`, `"MAE"` for regression; `"ACC"`, `"AUC"` for classification)
`geometry_method`	`"arf"`	Geometry scorer: `"arf"` or `"carf"`
`bins`	`10`	Number of quantile bins per axis in the 2D grid
`weak_fraction`	`0.2`	Fraction of worst-performing test bins flagged
`top_n_features`	`10`	Number of features for distribution plots
`min_count`	`20`	Minimum samples per bin to compute metric
`geometry_n_estimators`	`120`	Number of trees for CARF geometry scorer
`geometry_max_depth`	`12`	Max depth for CARF geometry scorer
`geometry_min_samples_leaf`	`20`	Min samples per leaf for CARF geometry scorer
`geometry_num_trees`	`30`	Number of trees for ARF geometry scorer
`geometry_max_iters`	`10`	Max adversarial iterations for ARF scorer
`mi_n_estimators`	`200`	Number of trees for MI scorer
`mi_max_depth`	`0`	Max depth for MI scorer (0 = unlimited)
`mi_min_samples_leaf`	`10`	Min samples per leaf for MI scorer
`mi_n_splits`	`5`	Number of cross-validation splits for MI scorer
`random_state`	`0`	Random seed for reproducibility

Key Method

TestSuite.diagnose_weakness_region()

See the TestSuite API Reference for full method documentation.