{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n# Weak-Cluster Detection and Repair (FuseKernel + MoE)\n\nDetect the weak regions of a model with FuseKernel, then repair them with a\nresidual-driven Mixture of Experts. FuseKernel clusters samples in its learned\nkernel geometry and breaks performance down per cluster (automatic weakness\ndetection); ``MoMoERegressor`` with ``cluster_method=\"ltc\"`` then fits a\nspecialised expert per learning-trajectory cluster, improving accuracy most where\nthe base model was weakest.\n\nFor fast detection the FuseKernel uses only the XGBoost tree co-membership kernel\n(``use_rbf=False``, ``use_spectral=False``) at the base model's depth, decoded with\nthe Nystrom solver -- no dense RBF or spectral fit is needed just to locate the\nweak clusters. A subsample of CaliforniaHousing keeps the example quick.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Installation\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# To install the required package, use the following command:\n# !pip install modeva" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Authentication\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# To get authentication, use the following command: (To get full access please replace the token to your own token)\n# from modeva.utils.authenticate import authenticate\n# authenticate(auth_code='eaaa4301-b140-484c-8e93-f9f633c8bacb')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Import required modules\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport numpy as np\nimport pandas as pd\nfrom sklearn.metrics import r2_score\nfrom modeva import DataSet\nfrom modeva.models import MoXGBRegressor, MoFuseKernelRegressor, MoMoERegressor" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Load and subsample the dataset\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "ds = DataSet()\nds.load(name=\"CaliforniaHousing\")\nds.set_active_samples()\nsub = ds.subsample_random(dataset=\"main\", sample_size=3000)\nds.set_active_samples(dataset=\"main\", sample_idx=sub.value[\"sample_idx\"])\nds.set_random_split()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Base model\nA depth-3 XGBoost whose weak regions we will detect and repair.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "base = MoXGBRegressor(name=\"base\", max_depth=3, n_estimators=100,\n learning_rate=0.1, random_state=0)\nbase.fit(ds.train_x, ds.train_y.ravel())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Detect weak clusters with FuseKernel\nFuseKernel clusters samples in its learned kernel geometry; ``diagnose_weak_clusters``\nbreaks train/test performance down per cluster so the weak regions surface\nautomatically. For fast detection we use only the tree co-membership kernel at the\nbase model's depth (``use_rbf=False``, ``use_spectral=False``), decoded with Nystrom.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "fk = MoFuseKernelRegressor(name=\"FuseKernel\", use_xgb=True, use_rbf=False,\n use_spectral=False, fit_method=\"grid\", solver=\"nystrom\",\n gbdt_params={\"n_estimators\": 100, \"max_depth\": 3})\nfk.fit(ds.train_x, ds.train_y.ravel())\n\ndiag = fk.diagnose_weak_clusters(ds, n_clusters=6)\ndiag.table" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The weakest cluster and its test-row mask.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "worst = diag.value[\"worst_clusters\"]\nworst_id = int(worst.iloc[0][\"cluster\"])\nweak_mask = np.asarray(diag.value[\"labels_test\"]) == worst_id\nprint(f\"weak cluster = {worst_id}, test samples in it = {int(weak_mask.sum())}\")\nworst" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Repair with a residual-driven Mixture of Experts\n``MoMoERegressor`` with ``cluster_method=\"ltc\"`` fits a baseline, clusters samples\nby their learning trajectories (where the base model struggles), and fits a\nspecialised expert per cluster with a gate routing between them.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "moe = MoMoERegressor(name=\"MoE-repair\", n_clusters=5, cluster_method=\"ltc\",\n expert=\"xgboost\", max_depth=3, n_estimators=100, random_state=0)\nmoe.fit(ds.train_x, ds.train_y.ravel())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Before vs after\nThe repair helps overall, and most on the FuseKernel-detected weak cluster.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "Xte, yte = np.asarray(ds.test_x), np.asarray(ds.test_y).ravel()\n\ndef r2(model, mask=None):\n p = model.predict(Xte)\n if mask is None:\n mask = np.ones(len(yte), dtype=bool)\n return round(r2_score(yte[mask], p[mask]), 4)\n\npd.DataFrame({\n (\"overall\", \"base\"): {\"R2\": r2(base)},\n (\"overall\", \"MoE repair\"): {\"R2\": r2(moe)},\n (f\"weak cluster {worst_id}\", \"base\"): {\"R2\": r2(base, weak_mask)},\n (f\"weak cluster {worst_id}\", \"MoE repair\"): {\"R2\": r2(moe, weak_mask)},\n}).T" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.11" } }, "nbformat": 4, "nbformat_minor": 0 }