{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Weakness Region Analysis (Classification)\n\nThis example demonstrates AMIF (Adversarial Mutual Information Forest) weakness\nregion diagnostics for classification models. In addition to the regression\ndiagnostics, classification includes confusion matrix comparisons between the\nfull test set and weak regions.\n\nWe use the TaiwanCredit dataset with an XGBoost classifier.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Setup\nImport libraries and suppress warnings.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import warnings\nwarnings.filterwarnings(\"ignore\")\n\nfrom modeva import DataSet, TestSuite\nfrom modeva.models import MoXGBClassifier"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Load Dataset\nLoad the TaiwanCredit dataset and set up for classification.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "ds = DataSet()\nds.load(\"TaiwanCredit\")\nds.set_target(feature=\"FlagDefault\")\nds.set_task_type(\"Classification\")\nds.set_random_split(test_ratio=0.2)\n\nprint(f\"Train: {ds.train_x.shape}, Test: {ds.test_x.shape}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Train Model\nTrain an XGBoost classifier.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "model = MoXGBClassifier(\n    name=\"XGBoost\", n_estimators=200, max_depth=4, learning_rate=0.1\n)\nmodel.fit(ds.train_x, ds.train_y.ravel())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Run Weakness Region Diagnostics\nUse ``diagnose_weakness_region`` with the CARF geometry method and ACC metric.\nThe ``min_count`` parameter sets the minimum number of samples required\nper region bin to compute a valid metric.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "ts = TestSuite(ds, model)\n\nresult = ts.diagnose_weakness_region(\n    geometry_method=\"carf\",\n    metric=\"ACC\",\n    bins=8,\n    weak_fraction=0.2,\n    top_n_features=10,\n    min_count=15,\n)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Result Table\nThe result table summarizes performance across all 2D grid regions.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result.table"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Print the weak region summary statistics.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "v = result.value\nprint(f\"Weak test samples: {v['n_weak_samples']} / {v['n_total_samples']} \"\n      f\"({100*v['n_weak_samples']/v['n_total_samples']:.1f}%)\")\nprint(f\"Metric: {v['metric']}, Cutoff: {v['cutoff']:.4f}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Region Performance Heatmaps\nHeatmaps show model performance across the 2D geometry-MI grid.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result.plot(name=\"region_performance_train\", figsize=(6.5, 5))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Test performance heatmap.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result.plot(name=\"region_performance_test\", figsize=(6.5, 5))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Sample count per region.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result.plot(name=\"region_sample_count\", figsize=(6.5, 5))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Feature Rankings\nJS divergence ranking shows which features differ most between weak and\nnon-weak regions.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result.plot(name=\"js_divergence_ranking\", figsize=(6.5, 4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "MI importance ranking.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result.plot(name=\"mi_importance_ranking\", figsize=(6.5, 4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Score Distributions\nDistribution of geometry and MI scores across the dataset.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result.plot(name=\"geometry_score_distribution\", figsize=(6.5, 4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "MI score distribution.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result.plot(name=\"mi_score_distribution\", figsize=(6.5, 4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Confusion Matrices\nCompare confusion matrices between the full test set and the weak region.\nThis reveals whether certain classes are disproportionately affected.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result.plot(name=(\"confusion_matrix\", \"all_test\"), figsize=(5, 4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Confusion matrix for weak region samples only.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result.plot(name=(\"confusion_matrix\", \"weak_test\"), figsize=(5, 4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Feature Distributions (Weak vs. Rest)\nCompare feature distributions between weak and non-weak samples\nfor the top 3 most divergent features.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "feat_figs = [\n    f for f in result.get_figure_names()\n    if isinstance(f, tuple) and f[0] == \"feature_distribution\"\n]\nfor fig_name in feat_figs[:3]:\n    result.plot(name=fig_name, figsize=(6.5, 4))"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.11"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}