Error-Aware Clustering

Overview

Error-aware clustering groups samples by their residual behavior rather than by raw feature proximity, so segments where a model makes similar (large) errors are surfaced as coherent clusters. Where slicing partitions one or two features at a time, this method clusters over the full feature space through a supervised model fit to the residual, exposing multivariate pockets of weak performance.

Two clustering algorithms are available, both operating on a residual response:

• LTC (Learning Trajectory Cluster) — the default. A gradient-boosting model is fit to the residual response; its per-stage prediction trajectories are weighted, reduced with PCA, and clustered with k-means. Samples that the booster learns along a similar path fall in the same cluster.

• RF (Random Forest) — a random forest is fit to the residual response; a proximity matrix is built from how often samples share tree leaves, and the resulting distance matrix is clustered with KMedoids.

Method

The test proceeds in three steps:

1. Residual response: For each sample a residual-based response is computed. The response type is configurable (response_type):

   • abs_residual / sq_residual — absolute or squared residual (default abs_residual).

   • abs_residual_perturb / sq_residual_perturb — residual after input perturbation, as in the robustness test.

   • pi_width — conformal prediction-interval width, as in the reliability test.

2. Clustering: Samples are grouped into n_clusters clusters using LTC or RF (cluster_method).

3. Per-cluster evaluation: The model metric and average residual are computed per cluster, ranking clusters from best to worst so weak segments stand out.

Implementation in MoDeVa

from modeva import DataSet, TestSuite
from modeva.models import MoXGBRegressor

# Load data
ds = DataSet()
ds.load("CaliforniaHousing")
ds.set_target(feature="MedHouseVal")
ds.set_task_type("Regression")
ds.set_random_split(test_ratio=0.2)

# Train model
model = MoXGBRegressor(
    name="XGBoost", n_estimators=200, max_depth=5, learning_rate=0.1
)
model.fit(ds.train_x, ds.train_y.ravel())

# Run error-aware clustering
ts = TestSuite(ds, model)
result = ts.diagnose_residual_cluster(
    dataset="test",
    response_type="abs_residual",
    n_clusters=10,
    cluster_method="ltc",   # or "rf"
)

# Per-cluster performance table
result.table

Output Interpretation

diagnose_residual_cluster returns a ValidationResult whose table ranks the clusters by performance and whose value holds the cluster labels, the assignment function, and per-cluster sample indices. Three plots are available.

Cluster Residual

Average residual response per cluster — the clusters with the highest bars are the weak segments the model struggles with most.

result.plot(name="cluster_residual")

Cluster Performance

The model metric per cluster, making performance heterogeneity across regions directly visible.

result.plot(name="cluster_performance")

Feature Importance

Feature importance of the random forest fit to the residual response, indicating which features most explain where the model errs.

result.plot(name="feature_importance")

Drilling into a Weak Cluster

Each cluster carries the sample indices inside and outside it, which can be fed straight into a data-distribution test to characterize how a weak cluster differs from the rest:

data_results = ds.data_drift_test(**result.value["clusters"][2]["data_info"])
data_results.plot("summary")
data_results.plot(("density", "MedInc"))

Parameters

The diagnose_residual_cluster method accepts the following key parameters:

For the perturbation response types, the robustness-test perturbation parameters (perturb_features, perturb_method, noise_level, n_repeats) also apply; for pi_width, the conformal alpha applies.

Key Method

• TestSuite.diagnose_residual_cluster()

See the TestSuite API Reference for full method documentation.