{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# ICL-MoE Classification\n\nThis example demonstrates ICL-MoE post-processing on a fitted DirectRS\nclassifier. ICL-MoE operates in logit space internally, adding kNN-based\nresidual correction on top of DirectRS leaf experts. The ``predict_proba``\nmethod applies sigmoid to produce calibrated probabilities.\n\nWe use the TaiwanCredit dataset with a depth-2 XGBoost base model.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Setup\nImport libraries and suppress warnings.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport numpy as np\nfrom modeva import DataSet, TestSuite\nfrom modeva.models import MoXGBClassifier"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Load Dataset\nLoad the TaiwanCredit dataset and create a random train/test split.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "ds = DataSet()\nds.load(name=\"TaiwanCredit\")\nds.set_random_split()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Train Base Model\nTrain an XGBoost classifier with depth 2.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "model = MoXGBClassifier(\n    name=\"XGB-cls-depth2\",\n    n_estimators=200, max_depth=2, learning_rate=0.1,\n    subsample=0.8, colsample_bytree=0.8,\n    random_state=42, verbosity=0\n)\nmodel.fit(ds.train_x, ds.train_y.ravel())\n\nts = TestSuite(ds, model)\nts.diagnose_accuracy_table().table"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Fit DirectRS\nPost-process the trained XGBoost classifier with DirectRS.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from modeva.models import MoDirectRSClassifier\n\ndrs = MoDirectRSClassifier(\n    base_model=model, ridge_alpha=100.0, n_passes=1\n)\ndrs.fit(ds.train_x, ds.train_y.ravel(), verbose=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Fit ICL-MoE\nBuild ICL-MoE on top of the fitted DirectRS classifier. The hierarchical\nvariant combines leaf expert predictions with a kNN residual correction.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from modeva.models import MoDirectRSICLClassifier\n\nicl = MoDirectRSICLClassifier(\n    directrs_model=drs, k=50, tau=1.0, ridge_lambda=1.0\n)\nicl.fit(ds.train_x, ds.train_y.ravel(), verbose=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Accuracy Comparison\nCompare AUC and accuracy between XGBoost, DirectRS, and ICL-MoE.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import roc_auc_score, accuracy_score, log_loss\n\ny_test = ds.test_y.ravel()\n\nbase_proba = model.predict_proba(ds.test_x)[:, 1]\ndrs_proba = drs.predict_proba(ds.test_x)[:, 1]\nicl_proba = icl.predict_proba(ds.test_x)[:, 1]\n\nprint(f\"{'Metric':<10s} {'XGBoost':>10s} {'DirectRS':>10s} {'ICL-MoE':>10s}\")\nprint(\"-\" * 42)\nprint(f\"{'AUC':<10s} {roc_auc_score(y_test, base_proba):>10.4f} \"\n      f\"{roc_auc_score(y_test, drs_proba):>10.4f} \"\n      f\"{roc_auc_score(y_test, icl_proba):>10.4f}\")\nprint(f\"{'Accuracy':<10s} {accuracy_score(y_test, model.predict(ds.test_x)):>10.4f} \"\n      f\"{accuracy_score(y_test, drs.predict(ds.test_x)):>10.4f} \"\n      f\"{accuracy_score(y_test, icl.predict(ds.test_x)):>10.4f}\")\nprint(f\"{'LogLoss':<10s} {log_loss(y_test, np.clip(base_proba, 1e-7, 1-1e-7)):>10.4f} \"\n      f\"{log_loss(y_test, np.clip(drs_proba, 1e-7, 1-1e-7)):>10.4f} \"\n      f\"{log_loss(y_test, np.clip(icl_proba, 1e-7, 1-1e-7)):>10.4f}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Probability Sanity Check\nVerify ICL-MoE probabilities are valid (in [0, 1], rows sum to 1).\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "proba = icl.predict_proba(ds.test_x)\n\nprint(f\"Shape: {proba.shape}\")\nprint(f\"Range: [{proba.min():.6f}, {proba.max():.6f}]\")\nprint(f\"Row sums: [{proba.sum(axis=1).min():.10f}, {proba.sum(axis=1).max():.10f}]\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Local Explanation\nFor classification, the decomposition operates on raw scores (logits).\nWe verify using the internal core which returns raw logits.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result = icl.explain_local(ds.test_x, feature_names=ds.feature_names)\nlocal = result.value\n\nraw_logits = icl._icl_core.predict(ds.test_x)\nrecon = local['intercept'] + local['contributions'].sum(axis=1)\nmax_err = np.max(np.abs(raw_logits - recon))\n\nprint(f\"Max |logit - (intercept + sum contributions)|: {max_err:.2e}\")\nprint(f\"Decomposition exact to machine precision: {max_err < 1e-10}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Local explanation waterfall plot for sample 0.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result.plot()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Global Feature Importance\nCompute global feature importance using mean absolute contributions.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result = icl.importance_global(\n    ds.test_x, feature_names=ds.feature_names, mode=\"contrib_abs\"\n)\nresult.plot()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Importance Comparison\nCompare ICL-MoE, DirectRS, and FANOVA feature importance side-by-side.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "icl_imp = icl.importance_global(\n    ds.test_x, feature_names=ds.feature_names, mode=\"contrib_abs\"\n).value['importance']\n\ndrs_imp = drs.importance_global(\n    ds.test_x, feature_names=ds.feature_names, mode=\"contrib_abs\"\n).value['importance']\n\nresult_fanova = ts.interpret_fi()\nfanova_table = result_fanova.table\nfanova_imp = dict(zip(fanova_table[\"Name\"], fanova_table[\"Score\"]))\n\nprint(f\"{'Feature':<16s} {'ICL-MoE':>10s} {'DirectRS':>10s} {'FANOVA':>10s}\")\nprint(\"-\" * 48)\nfor i, feat in enumerate(ds.feature_names):\n    print(f\"{feat:<16s} {icl_imp[i]:>10.4f} {drs_imp[i]:>10.4f} {fanova_imp.get(feat, 0.0):>10.4f}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "FANOVA feature importance plot.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result_fanova.plot()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "FANOVA interaction importance plot.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result_ei = ts.interpret_ei()\nresult_ei.plot()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Neighbour Analysis\nInspect the kNN neighbourhood used for residual correction.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result = icl.get_neighbor_analysis(\n    ds.test_x, sample_index=0, feature_names=ds.feature_names\n)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Neighbour weights.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result.plot(\"weight_bar\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "PCA scatter of neighbours in z-space.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result.plot(\"neighbor_scatter\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Leaf Gating Analysis\nExamine per-leaf routing for a specific tree.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result = icl.get_leaf_gating_analysis(\n    ds.test_x, sample_index=0, tree_index=0,\n    feature_names=ds.feature_names\n)\n\nactive = result.value['active_leaf']\nprint(f\"Active leaf: {active}\")\nprint(f\"Tree 0 contribution: {result.value['prediction']:.4f}\")\nprint(f\"Number of leaves: {len(result.value['leaf_weights'])}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Leaf gating weights for tree 0.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result.plot()"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.11"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}