FuseKernel

Overview

FuseKernel is a single kernel-ridge model that fuses complementary kernel channels and decodes them with kernel ridge regression. Three channels are available and can be combined freely:

• a tree co-membership kernel from a gradient-boosted ensemble (XGBoost / LightGBM / CatBoost), at one depth or several depths fused together,

• an RBF kernel on the standardized features,

• a learned multi-scale spectral kernel (MS-SKM) that learns its own spectral density and captures periodic and high-order structure the other channels miss.

The fused predictor is a convex mixture of the chosen channels:

\[\hat f(\mathbf{x}) = \sum_c w_c \, K_c(\mathbf{x}, \mathcal{S}) \, (\mathbf{K}_{\mathcal{S}\mathcal{S}} + \lambda \mathbf{I})^{-1} \mathbf{y}_{\mathcal{S}}\]

where the channel weights \(w_c\) are convex (\(w_c \ge 0\), \(\sum_c w_c = 1\)) and \(\lambda\) is the ridge. Every channel has a unit diagonal, so the mixture stays unit-diagonal and \(\lambda\) is identifiable.

Model Architecture

FuseKernel fits and selects the model in a leakage-free way:

1. Support / query split. The training data is split once into a support fold and a held-out query fold.

2. Channels fit on the support only. The GBDT backend and the MS-SKM front-end are trained on the support fold, so the query fold is genuinely unseen by every channel.

3. Leakage-free weight selection. The channel weights \(w_c\) and ridge \(\lambda\) are selected on the held-out query. This matters because the tree co-membership kernel near-interpolates the rows its trees were fit on, so weights chosen in-sample over-credit it. Choosing them on the query credits a channel only for data it never trained on.

4. Inherent Gaussian-process structure. The fused kernel decoded by KRR is a GP, so the posterior gives a closed-form predictive variance — uncertainty with no second model.

Implementation in MoDeVa

from modeva import DataSet, ModelZoo, TestSuite
from modeva.models import MoFuseKernelRegressor

# Load data
ds = DataSet()
ds.load(name="CaliforniaHousing")
ds.set_random_split()

# Fuse the tree co-membership and learned spectral kernels
model = MoFuseKernelRegressor(
    name="FuseKernel",
    use_xgb=True, use_spectral=True,        # channels to fuse
    fit_method="grid",                       # leakage-free, query-scored
    spectral_params={"H": 4, "K": 8, "kernel": "laplace"},
    random_state=0,
)

mz = ModelZoo(dataset=ds)
mz.add_model(model)
mz.train_all()
mz.leaderboard()

The spectral channel’s kernel-ridge decode is controlled by the solver argument (only used when use_spectral=True). The default "nystrom" is the linear-in-n, low-rank decode — the fastest and most scalable choice, suitable for large datasets. "lanczos" is the exact dense decode (best for small data), "auto" uses dense below ~20k rows and switches to Nystrom above, and "matfree" is a matrix-free conjugate-gradient solve:

model = MoFuseKernelRegressor(
    name="FuseKernel", use_xgb=True, use_spectral=True,
    solver="nystrom",   # "nystrom" (default) | "auto" | "lanczos" | "matfree"
)

Once trained, the model works with the entire TestSuite — accuracy, robustness, reliability, resilience, fairness, slicing, and the post-hoc explainers (SHAP, LIME, PDP, ALE, PFI):

ts = TestSuite(dataset=ds, model=model)
ts.diagnose_accuracy_table().table
ts.diagnose_robustness().plot()

Classification

The same model handles classification: the fused KRR is fit on one-hot targets and probabilities are read off a temperature-calibrated softmax. Binary and multiclass are both supported, and the simplex weights are selected on the held-out query by accuracy.

from modeva.models import MoFuseKernelClassifier

ds = DataSet()
ds.load(name="TaiwanCredit")
ds.set_random_split()

model = MoFuseKernelClassifier(
    name="FuseKernel-Cls", use_xgb=True, use_spectral=True,
)
mz = ModelZoo(dataset=ds)
mz.add_model(model)
mz.train_all()

ts = TestSuite(dataset=ds, model=model)
ts.diagnose_accuracy_table().table       # AUC / ACC / F1 / LogLoss / Brier ...

Inherent Interpretation (FANOVA)

When a spectral channel is present (use_spectral=True), FuseKernel exposes an inherent functional-ANOVA decomposition of the fused predictor through the standard interpret_* methods. Main effects and pairwise interactions are computed from the model’s own predictions; interaction candidates are ranked by the spectral channel’s ARD relevance.

ts = TestSuite(dataset=ds, model=model)

ts.interpret_fi().plot()                  # feature importance from effect variance
ts.interpret_ei().plot()                  # effect importance (main + interactions)
ts.interpret_effects().plot("MedInc")     # main-effect curve for a feature
ts.interpret_local_fi(sample_index=0).plot()

Channel Decomposition

The fused prediction is an exact additive sum over channels, \(\hat f(\mathbf{x}) = f_0 + \sum_c w_c\, [K_c(\mathbf{x}, \mathcal{S})\,\boldsymbol\alpha]\). channel_contributions returns each channel’s contribution in target units, so you can see how much the tree, RBF, and spectral channels each drive a prediction (regression).

result = model.channel_contributions(ds.test_x)
result.table        # mean absolute contribution per channel
result.plot()

Inherent Predictive Uncertainty

For regression, FuseKernel’s prediction intervals are the closed-form Gaussian-process posterior of the fused kernel rather than a conformal wrapper, so diagnose_reliability reflects the model’s own uncertainty:

\[\mathrm{Var}(\mathbf{x}^*) = 1 - K(\mathbf{x}^*, \mathcal{S}) (\mathbf{K}_{\mathcal{S}\mathcal{S}} + \lambda \mathbf{I})^{-1} K(\mathcal{S}, \mathbf{x}^*) + \lambda .\]

ts = TestSuite(dataset=ds, model=model)
ts.diagnose_reliability().table           # avg width and coverage of the GP intervals

# or directly on the model
mean, var = model.predict_dist(ds.test_x)
lo_hi = model.predict_interval(ds.test_x)  # (n, 2) at the calibrated level

Diagnose and Repair

Because FuseKernel is a standard MoDeVa model, all single-model diagnostics and the multi-model comparison apply, and weak regions can be localized with slicing:

ts.diagnose_slicing_accuracy(features="MedInc").plot()
ts.diagnose_resilience().plot()

Beyond slicing, FuseKernel can detect weak regions using its own kernel geometry: diagnose_weak_clusters clusters the data in the fitted kernel’s Nyström spectral embedding and breaks train/test performance down per cluster, ranking the weakest ones.

diag = model.diagnose_weak_clusters(ds, n_clusters=6)
diag.table                       # per-cluster train/test breakdown
diag.value["worst_clusters"]     # the weakest clusters by test metric

The detected weak region is then repaired with a residual-driven Mixture of Experts (MoMoERegressor / MoMoEClassifier), which clusters samples by their learning trajectories and fits a specialised expert per cluster — optionally carrying the same monotonicity constraints so the repaired model stays interpretable:

from modeva.models import MoMoERegressor

moe = MoMoERegressor(name="MoE-repair", n_clusters=5, cluster_method="ltc",
                     expert="xgboost", max_depth=2)
moe.fit(ds.train_x, ds.train_y.ravel())

See Weak-Cluster Detection and Repair in the Diagnostic Suite for the full detect-and-repair workflow.

Key Classes

• MoFuseKernelRegressor — FuseKernel for regression tasks

• MoFuseKernelClassifier — FuseKernel for classification tasks

See the API Reference for full method documentation.