GBDT Leaf Kernel

Overview

The GBDT leaf kernel reinterprets a fitted gradient-boosted tree ensemble as a generalized Nadaraya-Watson (GNW) operator. Every prediction of a boosted ensemble can be written as a kernel-weighted average of training responses, where the kernel is the ensemble’s own leaf co-membership: two inputs are close when the trees route them to the same leaves. This turns an opaque GBDT into a nearest-neighbor model over a learned geometry, which exposes the evidence behind each prediction without a separate surrogate.

The wrapper takes a pre-trained XGBoost model and re-uses its leaf geometry, so it does not retrain the ensemble. On the same geometry it offers five interchangeable prediction heads, from an exact reconstruction of the base model to smoothed and repaired variants.

Prediction Heads

Let $K(\mathbf{x}, \mathbf{x}_i)$ be the leaf co-membership kernel of the fitted ensemble and let $w_i \propto K(\mathbf{x}, \mathbf{x}_i)^{\rho}$ be the row-normalized weights over the retrieved neighbors. The five heads share this geometry and differ only in what value they average:

exact_gbdt (default) — the per-tree hard leaf selector with the fitted leaf scores as values. It reconstructs the base GBDT prediction exactly, so wrapping the model changes nothing until a different head is requested.
$\hat f_{\text{exact}}(\mathbf{x}) = \sum_{t} \gamma_{t,\, \ell_t(\mathbf{x})} = f_M(\mathbf{x}),$
where $\ell_t(\mathbf{x})$ is the leaf of tree $t$ and $\gamma_{t,\ell}$ its leaf score.
gnw_label — the row-normalized leaf kernel with the training labels as values, $V_i = y_i$ . A local label smoother: $\hat f(\mathbf{x}) = \sum_i w_i\, y_i$ .
gnw_leaf — the same kernel with the base model’s fitted values, $V_i = f_M(\mathbf{x}_i)$ . A smoothed version of the base predictor.
krr — kernel ridge regression on the leaf one-hot basis, regularized by ridge_lambda.
residual_gnw — the base model plus a cluster-gated Nadaraya-Watson correction of out-of-fold residuals. A repair head that adjusts the base prediction only in behavioral cohorts where the residuals are significant.

The head is chosen at prediction time:

model.predict(X, head="gnw_label")     # regression
model.predict_proba(X, head="krr")     # classification

Implementation in MoDeVa

The wrapper is fitted on a base model that has already been trained. A held-out validation fold (X_val, y_val) calibrates the cluster gate and supports head nomination.

from sklearn.model_selection import train_test_split
from modeva import DataSet
from modeva.models import MoXGBRegressor, MoGBDTKernelRegressor

# Load data
ds = DataSet()
ds.load(name="CaliforniaHousing")
ds.set_target("MedHouseVal")
ds.preprocess()
ds.set_random_split(test_ratio=0.2, random_state=0)

X_tr, X_val, y_tr, y_val = train_test_split(
    ds.train_x, ds.train_y.ravel(), test_size=0.2, random_state=0)

# 1. Train the base GBDT
base = MoXGBRegressor(name="XGB-base", n_estimators=200, max_depth=4,
                      learning_rate=0.1, random_state=0)
base.fit(X_tr, y_tr)

# 2. Wrap its leaf geometry as a GNW operator
model = MoGBDTKernelRegressor(base_model=base, name="LK-XGB",
                              kernel_topk=200, kernel_rho=1.0,
                              ridge_lambda=1.0, n_clusters=8,
                              gate_mode="defensive", random_state=0)
model.fit(X_tr, y_tr, X_val=X_val, y_val=y_val,
          feature_names=ds.feature_names)

# Exact reconstruction of the base model (Theorem 1)
import numpy as np
gap = np.max(np.abs(base.predict(ds.test_x).ravel()
                    - model.predict(ds.test_x, head="exact_gbdt")))
# gap is ~1e-6

Because the default head reconstructs the base GBDT, the wrapper predicts identically to the ensemble it wraps until another head is selected. This makes it a drop-in for downstream MoDeVa diagnostics that call predict / predict_proba.

Classification

The classifier wraps a fitted binary XGBoost classifier. All heads operate in logit space internally; predict_proba applies the sigmoid and decision_function returns the margin.

from modeva.models import MoXGBClassifier, MoGBDTKernelClassifier

ds = DataSet()
ds.load(name="SimuCredit")
ds.set_target("Status")
ds.preprocess()
ds.set_random_split(test_ratio=0.2, random_state=0)

X_tr, X_val, y_tr, y_val = train_test_split(
    ds.train_x, ds.train_y.ravel(), test_size=0.2, random_state=0)

base = MoXGBClassifier(name="XGB-clf", n_estimators=200, max_depth=4,
                       learning_rate=0.1, random_state=0)
base.fit(X_tr, y_tr)

model = MoGBDTKernelClassifier(base_model=base, name="LK-XGB-clf",
                               kernel_topk=200, gate_mode="defensive")
model.fit(X_tr, y_tr, X_val=X_val, y_val=y_val,
          feature_names=ds.feature_names)

proba = model.predict_proba(ds.test_x)               # default exact_gbdt head
margin = model.decision_function(ds.test_x, head="gnw_leaf")

Local Evidence (TKLE)

Every prediction is a weighted vote of retrieved training cases, so it can be audited case by case. explain_local returns the Tree Kernel Local Evidence ledger for a query: the neighbors that drove the prediction, their kernel weights, their labels and their base-model fitted values.

local = model.explain_local(ds.test_x[:1], sample_index=0,
                            feature_names=ds.feature_names)
local.table                        # top-20 neighbors: weight, K(x,x_i), y_i, f_M(x_i)
local.plot("weight_bar")           # neighbor weight bar
local.plot("neighbor_scatter")     # neighbors in the kernel neighborhood

Across a batch, get_evidence_diagnostics summarizes six TKLE quantities — including the effective neighbor count $N_{\text{eff}}$ and the label, query and kernel dispersions — so low-support or high-disagreement predictions can be flagged.

diag = model.get_evidence_diagnostics(ds.test_x)
diag.table                         # mean / median / p10 / p90 per diagnostic
diag.plot("neff")                  # effective-neighbor-count histogram

Head Nomination

Which head to trust is a diagnostic question, not a fixed choice. nominate_head scores every head on a validation fold and reports the region bias, region variance, kernel-ridge degrees of freedom and spectral concentration used to recommend one.

nom = model.nominate_head(X_val, y_val)
nom.table["errors"]                # per-head validation error, sorted
nom.table["diagnostics"]           # B_R, V_R, df(lambda), C(10), recommended head
nom.plot()

Global Importance and Weakness

importance_global ranks features by how tightly the retrieved neighborhoods concentrate along each feature under the learned geometry. diagnose_weakness partitions the data into behavioral cohorts and reports per-cluster loss together with a Jensen-Shannon feature profile that characterizes each weak cohort.

imp = model.importance_global(ds.test_x, feature_names=ds.feature_names)
imp.table
imp.plot()

weak = model.diagnose_weakness(top_features=8, n_bins=10)
weak.table["clusters"]             # per-cluster loss
weak.table["js"]                   # JS feature profile of weak cohorts
weak.plot()

In-Context Prediction

Because the base geometry is fixed by the fitted trees, the retrieval memory can be swapped without refitting. predict_with_context uses a supplied (context_X, context_y) pool as the Nadaraya-Watson memory, which answers counterfactual “what if these cases had been observed?” queries. Only the neighbor-based heads (gnw_label, gnw_leaf) admit an external context.

pred, weights, idx = model.predict_with_context(
    ds.test_x[:5], context_X=ds.train_x, context_y=ds.train_y.ravel(),
    head="gnw_label")
# weights, idx index the retrieved cases in the supplied context pool

Key Classes

~modeva.models.kernel_xgb.api.MoGBDTKernelRegressor — GBDT leaf kernel for regression tasks
~modeva.models.kernel_xgb.api.MoGBDTKernelClassifier — GBDT leaf kernel for binary classification tasks

See the API Reference </modules/gbdt_kernel> for full method documentation.