MoDeVa 2.0 — What's New

DirectRS

Tree ensembles are among the most accurate models for tabular data, and among the hardest to explain. DirectRS post-processes a pre-trained ensemble (XGBoost, LightGBM, CatBoost) without retraining it: it extracts a global stretch matrix from the tree structure, then fits a per-tree Ridge regression on the stretched embedding, replacing each tree's piecewise-constant leaf value with a piecewise-linear one.

The identity holds to machine precision — no sampling and no approximation, unlike post-hoc attribution. DirectRS supplies six interpretability methods: a global stretch analysis, exact local decomposition, four global-importance modes, a main-vs-interaction variance split, geometric interaction traces and a top pairwise-interaction table.

GBDT Leaf Kernel

The GBDT leaf kernel re-reads a fitted ensemble rather than reshaping it. Every gradient-boosted prediction can be written as a weighted average of training labels, scores or residuals under the ensemble's own leaf co-membership kernel — two inputs are close when the trees route them to the same leaves. This turns an opaque GBDT into a nearest-neighbor model over a learned geometry and exposes the evidence behind each prediction without a separate surrogate.

It offers five interchangeable prediction heads — exact_gbdt, gnw_label, gnw_leaf, krr and residual_gnw — with a nominate_head procedure that selects one on validation evidence. Every prediction carries a Top-k Leaf-Evidence (TKLE) ledger, and predict_with_context swaps the training memory for a user-supplied pool while keeping the kernel fixed, which supports counterfactual and case-based audits without retraining.

FuseKernel

A single kernel encodes one prior over functions, and no one kernel is right for every dataset. FuseKernel forms a convex mixture of up to three channels — a tree co-membership kernel, an RBF kernel and a learned multi-scale spectral kernel — and decodes the mixture with a single kernel ridge:

The mixture weights and the ridge are chosen on a held-out query fold that no channel has trained on, which keeps the fusion from over-crediting a channel that memorizes its own training rows. FuseKernel exposes an inherent functional-ANOVA interpretation and closed-form Gaussian-process prediction intervals for reliability diagnostics without a conformal wrapper.

ICL-MoE

ICL-MoE (In-Context Learning Mixture of Experts) builds on a fitted DirectRS model. Two samples that land near each other in the DirectRS embedding share similar split-routing patterns across all trees, so proximity there is structural similarity from the model's perspective rather than raw feature proximity. ICL-MoE replaces hard tree-based leaf routing with soft k-nearest-neighbor attention gating in that space: for each query it finds its contextual neighbors and fits a local expert on them, correcting the DirectRS leaf experts with a kNN-weighted local Ridge on their residuals. This is the tabular analogue of in-context learning, where the context is the set of nearby training examples in the learned embedding.

Error decomposition

A model with acceptable aggregate accuracy can fail in specific subpopulations. The error decomposition splits a single aggregate number into per-cluster contributions so the failure is visible. For classification the global AUC decomposes exactly into intra-cluster and inter-cluster blocks:

The diagonal terms measure discrimination within a cluster; the off-diagonal terms measure how the model ranks one cluster's positives against another's negatives. A low diagonal entry marks a subgroup the model cannot separate; a low off-diagonal entry marks cross-subgroup miscalibration, a fairness concern the global AUC hides.

For regression the within-cluster MSE decomposes into four interpretable terms — calibration, prediction variance, outcome variance and alignment:

This separates reducible error — poor calibration or misspent capacity — from the irreducible outcome variance that no model can beat in that region. Summing over clusters adds a between-cluster resolution block that measures how well the clustering separates distinct outcome levels. Under binary outcomes with constant within-cluster predictions this recovers Murphy's Brier decomposition exactly (reliability, resolution, uncertainty), so the framework generalizes the classical result to continuous outcomes.

The clustering that drives the decomposition is itself error-aware. Error-Aware Random Forest (EARF) clustering trains a random forest to predict residual magnitude from the features and clusters samples by the forest's leaf co-occurrence, so samples are grouped by shared error structure rather than feature similarity. The clusters can span non-contiguous feature regions, precisely because error patterns cut across geometric boundaries.

A dedicated User Guide page for the error decomposition is still to come. In the meantime this note is the reference for the method.

Weakness detection and repair with FuseKernel

FuseKernel adds a way to find a model's weak regions and repair them. The detection step uses the fused kernel as a learned geometry: diagnose_weak_clusters embeds the fitted kernel with a Nyström spectral map, partitions the data into clusters in that geometry and reports the model metric per cluster on train and test. Breaking performance down per cluster surfaces the weak regions without having to guess where to look; the result ranks the worst clusters and maps each test row to its cluster. Detection is cheap — only the tree co-membership kernel at the base model's depth is needed, so the RBF and spectral channels can stay off.

The repair step fits a residual-driven Mixture of Experts (MoMoERegressor / MoMoEClassifier). With cluster_method="ltc" it clusters samples by their learning trajectories — where the base model struggles — fits a specialized expert per cluster and routes between them with a gate. The experts can carry the same monotonicity constraints as the base model, so the repaired model stays interpretable. The gain is largest where the base model was weakest.

Weakness diagnostics on the GBDT leaf kernel

The GBDT leaf kernel carries its own weakness diagnostic. diagnose_weakness ranks the ensemble's behavioral clusters by their contribution to validation loss and, for the weakest cluster, identifies the features that distinguish it from the rest by Jensen–Shannon divergence between the in-cluster and out-of-cluster feature histograms. The features with the largest divergence name the weak case — they describe the kind of case the ensemble currently struggles with. This is residual weakness slicing anchored in the model's own leaf kernel rather than in raw feature axes.

AMIF weakness-region detection

TestSuite.diagnose_weakness_region locates weak regions of the input space with an Adversarial Mutual Information Forest (AMIF) pipeline: geometry scoring via an adversarial random forest, mutual-information scoring via a cross-validated random forest, then 2D binning with a configurable grid and a weak-fraction cutoff. Features are ranked by Jensen–Shannon divergence between the weak and non-weak regions. The outputs include region-performance heatmaps for train and test, divergence and mutual-information rankings, score distributions and per-feature distributions, and confusion matrices for classification.