{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Weakness Region Analysis (Regression)\n\nThis example demonstrates AMIF (Adversarial Mutual Information Forest) weakness\nregion diagnostics for regression models. AMIF identifies model weakness regions\nby partitioning data into a 2D grid of geometry scores (data density via CARF)\nand MI scores (predictive information), then finding regions where the model\nperforms worst.\n\nWe use the CaliforniaHousing dataset with an XGBoost regressor.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Setup\nImport libraries and suppress warnings.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import warnings\nwarnings.filterwarnings(\"ignore\")\n\nfrom modeva import DataSet, TestSuite\nfrom modeva.models import MoXGBRegressor"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Load Dataset\nLoad the CaliforniaHousing dataset and set up for regression.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "ds = DataSet()\nds.load(\"CaliforniaHousing\")\nds.set_target(feature=\"MedHouseVal\")\nds.set_task_type(\"Regression\")\nds.set_random_split(test_ratio=0.2)\n\nprint(f\"Train: {ds.train_x.shape}, Test: {ds.test_x.shape}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Train Model\nTrain an XGBoost regressor.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "model = MoXGBRegressor(\n    name=\"XGBoost\", n_estimators=200, max_depth=5, learning_rate=0.1\n)\nmodel.fit(ds.train_x, ds.train_y.ravel())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Run Weakness Region Diagnostics\nUse ``diagnose_weakness_region`` with the CARF geometry method and MSE metric.\nThe ``weak_fraction`` parameter controls what fraction of the worst-performing\nregions are flagged as weak.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "ts = TestSuite(ds, model)\n\nresult = ts.diagnose_weakness_region(\n    geometry_method=\"carf\",\n    metric=\"MSE\",\n    bins=8,\n    weak_fraction=0.2,\n    top_n_features=8,\n)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Result Table\nThe result table summarizes performance across all 2D grid regions.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result.table"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Print the weak region summary statistics.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "v = result.value\nprint(f\"Weak test samples: {v['n_weak_samples']} / {v['n_total_samples']} \"\n      f\"({100*v['n_weak_samples']/v['n_total_samples']:.1f}%)\")\nprint(f\"Metric: {v['metric']}, Cutoff: {v['cutoff']:.4f}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Region Performance Heatmaps\nHeatmaps show model performance across the 2D geometry-MI grid.\nComparing train vs test performance reveals overfitting patterns.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result.plot(name=\"region_performance_train\", figsize=(6.5, 5))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Test performance heatmap.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result.plot(name=\"region_performance_test\", figsize=(6.5, 5))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Sample count per region.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result.plot(name=\"region_sample_count\", figsize=(6.5, 5))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Feature Rankings\nJS divergence ranking shows which features differ most between weak and\nnon-weak regions. MI importance ranking shows which features carry the\nmost predictive information.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result.plot(name=\"js_divergence_ranking\", figsize=(6.5, 4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "MI importance ranking.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result.plot(name=\"mi_importance_ranking\", figsize=(6.5, 4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Score Distributions\nDistribution of geometry and MI scores across the dataset.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result.plot(name=\"geometry_score_distribution\", figsize=(6.5, 4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "MI score distribution.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result.plot(name=\"mi_score_distribution\", figsize=(6.5, 4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Feature Distributions (Weak vs. Rest)\nCompare feature distributions between weak and non-weak samples\nfor the top 3 most divergent features.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "feat_figs = [\n    f for f in result.get_figure_names()\n    if isinstance(f, tuple) and f[0] == \"feature_distribution\"\n]\nfor fig_name in feat_figs[:3]:\n    result.plot(name=fig_name, figsize=(6.5, 4))"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.11"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}