FuseKernel

FuseKernel

Overview

FuseKernel is a single kernel-ridge model that fuses complementary kernel channels and decodes them with kernel ridge regression. Three channels are available and can be combined freely:

• a tree co-membership kernel from a gradient-boosted ensemble (XGBoost / LightGBM / CatBoost), at one depth or several depths fused together,
• an RBF kernel on the standardized features,
• a learned multi-scale spectral kernel (MS-SKM) that learns its own spectral density and captures periodic and high-order structure the other channels miss.

The fused predictor is a convex mixture of the chosen channels:

where the channel weights $w_c$ are convex ($w_c \ge 0$, $\sum_c w_c = 1$) and $\lambda$ is the ridge. Every channel has a unit diagonal, so the mixture stays unit-diagonal and $\lambda$ is identifiable.

Model Architecture

FuseKernel fits and selects the model in a leakage-free way:

1. Support / query split. The training data is split once into a support fold and a held-out query fold.
2. Channels fit on the support only. The GBDT backend and the MS-SKM front-end are trained on the support fold, so the query fold is genuinely unseen by every channel.
3. Leakage-free weight selection. The channel weights $w_c$ and ridge $\lambda$ are selected on the held-out query. This matters because the tree co-membership kernel near-interpolates the rows its trees were fit on, so weights chosen in-sample over-credit it. Choosing them on the query credits a channel only for data it never trained on.
4. Inherent Gaussian-process structure. The fused kernel decoded by KRR is a GP, so the posterior gives a closed-form predictive variance — uncertainty with no second model.

Implementation in MoDeVa

from modeva import DataSet, ModelZoo, TestSuite
from modeva.models import MoFuseKernelRegressor

# Load data
ds = DataSet()
ds.load(name="CaliforniaHousing")
ds.set_random_split()

# Fuse the tree co-membership and learned spectral kernels
model = MoFuseKernelRegressor(
    name="FuseKernel",
    use_xgb=True, use_spectral=True,        # channels to fuse
    fit_method="grid",                       # leakage-free, query-scored
    spectral_params={"H": 4, "K": 8, "kernel": "laplace"},
    random_state=0,
)

mz = ModelZoo(dataset=ds)
mz.add_model(model)
mz.train_all()
mz.leaderboard()

Once trained, the model works with the entire TestSuite — accuracy, robustness, reliability, resilience, fairness, slicing, and the post-hoc explainers (SHAP, LIME, PDP, ALE, PFI):

ts = TestSuite(dataset=ds, model=model)
ts.diagnose_accuracy_table().table
ts.diagnose_robustness().plot()

Classification

The same model handles classification: the fused KRR is fit on one-hot targets and probabilities are read off a temperature-calibrated softmax. Binary and multiclass are both supported, and the simplex weights are selected on the held-out query by accuracy.

from modeva.models import MoFuseKernelClassifier

ds = DataSet()
ds.load(name="TaiwanCredit")
ds.set_random_split()

model = MoFuseKernelClassifier(
    name="FuseKernel-Cls", use_xgb=True, use_spectral=True,
)
mz = ModelZoo(dataset=ds)
mz.add_model(model)
mz.train_all()

ts = TestSuite(dataset=ds, model=model)
ts.diagnose_accuracy_table().table       # AUC / ACC / F1 / LogLoss / Brier ...

Inherent Interpretation (FANOVA)

When a spectral channel is present (use_spectral=True), FuseKernel exposes an inherent functional-ANOVA decomposition of the fused predictor through the standard interpret_* methods. Main effects and pairwise interactions are computed from the model’s own predictions; interaction candidates are ranked by the spectral channel’s ARD relevance.

ts = TestSuite(dataset=ds, model=model)

ts.interpret_fi().plot()                  # feature importance from effect variance
ts.interpret_ei().plot()                  # effect importance (main + interactions)
ts.interpret_effects().plot("MedInc")     # main-effect curve for a feature
ts.interpret_local_fi(sample_index=0).plot()

Channel Decomposition

The fused prediction is an exact additive sum over channels, $\hat f(\mathbf{x}) = f_0 + \sum_c w_c\, [K_c(\mathbf{x}, \mathcal{S})\,\boldsymbol\alpha]$. channel_contributions returns each channel’s contribution in target units, so you can see how much the tree, RBF, and spectral channels each drive a prediction (regression).

result = model.channel_contributions(ds.test_x)
result.table        # mean absolute contribution per channel
result.plot()

Inherent Predictive Uncertainty

For regression, FuseKernel’s prediction intervals are the closed-form Gaussian-process posterior of the fused kernel rather than a conformal wrapper, so diagnose_reliability reflects the model’s own uncertainty:

ts = TestSuite(dataset=ds, model=model)
ts.diagnose_reliability().table           # avg width and coverage of the GP intervals

# or directly on the model
mean, var = model.predict_dist(ds.test_x)
lo_hi = model.predict_interval(ds.test_x)  # (n, 2) at the calibrated level

Diagnose and Repair

Because FuseKernel is a standard MoDeVa model, all single-model diagnostics and the multi-model comparison apply, and weak regions can be localized with slicing:

ts.diagnose_slicing_accuracy(features="MedInc").plot()
ts.diagnose_resilience().plot()

Key Classes

• ~modeva.models.fusekernel.api.MoFuseKernelRegressor — FuseKernel for regression tasks
• ~modeva.models.fusekernel.api.MoFuseKernelClassifier — FuseKernel for classification tasks

See the API Reference </modules/fusekernel> for full method documentation.