AMIF Weakness Region Diagnostics

AMIF Weakness Region Diagnostics

Overview

AMIF (Adversarial Mutual Information Forest) identifies model weakness regions by partitioning data into a 2D grid based on two independent scoring axes:

• Geometry score — data density estimated via Adversarial Random Forest (ARF) or Conditional ARF (CARF)
• MI score — predictive mutual information estimated via cross-validated Random Forest

Regions where the model performs worst are flagged as “weak,” enabling targeted diagnosis of where and why a model fails.

Method

The AMIF pipeline consists of four steps:

1. Geometry Scoring: An ARF or CARF model estimates each sample’s density score. Low-density regions correspond to sparse or out-of-distribution data.
2. MI Scoring: A Random Forest cross-validates predictions to estimate each sample’s predictive information. Low MI indicates features carry little information about the target in that region.
3. 2D Binning: Samples are placed into a grid of bins x bins regions based on their geometry and MI scores. Per-region metrics are computed.
4. Weak Region Identification: Regions with the worst weak_fraction of performance are flagged. Feature distributions are compared between weak and non-weak samples using Jensen-Shannon divergence.

Implementation in MoDeVa

from modeva import DataSet, TestSuite
from modeva.models import MoXGBRegressor

# Load data
ds = DataSet()
ds.load("CaliforniaHousing")
ds.set_target(feature="MedHouseVal")
ds.set_task_type("Regression")
ds.set_random_split(test_ratio=0.2)

# Train model
model = MoXGBRegressor(
    name="XGBoost", n_estimators=200, max_depth=5, learning_rate=0.1
)
model.fit(ds.train_x, ds.train_y.ravel())

# Run weakness region diagnostics
ts = TestSuite(ds, model)
result = ts.diagnose_weakness_region(
    geometry_method="carf",
    metric="MSE",
    bins=8,
    weak_fraction=0.2,
    top_n_features=8,
)

# View result summary
result.table

Output Interpretation

The diagnose_weakness_region method returns a ValidationResult with multiple plot types:

Region Performance Heatmaps

Three heatmaps over the 2D geometry-MI grid:

• region_performance_train — training metric per region
• region_performance_test — test metric per region
• region_sample_count — number of samples per region

Comparing train vs test heatmaps reveals overfitting patterns: regions with good training performance but poor test performance indicate overfitting.

result.plot(name="region_performance_train", figsize=(6.5, 5))
result.plot(name="region_performance_test", figsize=(6.5, 5))
result.plot(name="region_sample_count", figsize=(6.5, 5))

JS Divergence Ranking

Ranks features by Jensen-Shannon divergence between their distributions in weak vs non-weak regions. High JS divergence indicates the feature has a very different distribution in weak regions, making it a key driver of model weakness.

result.plot(name="js_divergence_ranking", figsize=(6.5, 4))

MI Importance Ranking

Ranks features by their mutual information with the target variable. Features with high MI are important predictors overall.

result.plot(name="mi_importance_ranking", figsize=(6.5, 4))

Score Distributions

Distribution plots for both scoring axes, showing how geometry and MI scores are spread across the dataset.

result.plot(name="geometry_score_distribution", figsize=(6.5, 4))
result.plot(name="mi_score_distribution", figsize=(6.5, 4))

Feature Distributions

Side-by-side comparison of feature distributions between weak and non-weak samples. Available for the top N features ranked by JS divergence.

feat_figs = [
    f for f in result.get_figure_names()
    if isinstance(f, tuple) and f[0] == "feature_distribution"
]
for fig_name in feat_figs[:3]:
    result.plot(name=fig_name, figsize=(6.5, 4))

Confusion Matrices (Classification Only)

For classification tasks, confusion matrices are available for the full test set and the weak region subset:

result.plot(name=("confusion_matrix", "all_test"), figsize=(5, 4))
result.plot(name=("confusion_matrix", "weak_test"), figsize=(5, 4))

Parameters

The diagnose_weakness_region method accepts the following parameters:

Key Method

• TestSuite.diagnose_weakness_region() <modeva.testsuite.local_testsuite.LocalTestSuite.diagnose_weakness_region>

See the TestSuite API Reference </modules/testsuite> for full method documentation.