{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Feature Engineering Basics\n\nThis example demonstrates the basic feature engineering workflow in MoDeVa:\nqueue transformers, execute them, and inspect the results. We use the\nCaliforniaHousing dataset with three unsupervised transformers.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Setup\nImport libraries and suppress warnings.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport numpy as np\nfrom modeva import DataSet"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Load Dataset\nLoad the CaliforniaHousing dataset and create a random train/test split.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "ds = DataSet()\nds.load(name=\"CaliforniaHousing\")\nds.set_random_split()\n\n# Keep a copy of the raw (un-engineered) data to demonstrate transforming\n# new data later; ds.to_df() reflects the engineered frame after execution.\nraw_df = ds.to_df()\n\nprint(f\"Features: {ds.feature_names}\")\nprint(f\"Number of features: {len(ds.feature_names)}\")\nprint(f\"Training samples: {ds.train_x.shape[0]}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Queue Feature Engineering Steps\nFeature engineering uses a deferred execution pattern: first queue\nthe transformers, then execute them all at once.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Bivector features: pairwise products x_i * x_j\nds.fe_bivector(max_features=20, selection=\"variance\")\n\n# Whitening: SVD-based whitening (GA Mahalanobis transform)\nds.fe_whitening(regularization=1e-8)\n\n# Density: RBF kernel density estimate per sample\nds.fe_density(bandwidth=\"median\", n_reference=200)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Execute Feature Engineering\nRun all queued steps. Each step fits on the data and appends new columns.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "ds.engineer_features()\n\nprint(f\"Features after engineering: {len(ds.all_feature_names)}\")\nprint(f\"New feature names (first 10): {ds.all_feature_names[:10]}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Inspect Results\nCheck the feature engineering status.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "fe = ds.get_feature_engineer()\nstatus = fe.get_status()\nprint(f\"Executed steps: {[s['name'] for s in status['executed_steps']]}\")\nprint(f\"Pending steps: {[s['name'] for s in status['pending_steps']]}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Transform New Data\nApply the fitted pipeline to the test set.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "test_df = raw_df.iloc[:5]\ntransformed = ds.transform_features(test_df)\nprint(f\"Transformed shape: {transformed.shape}\")\nprint(f\"Columns: {list(transformed.columns[:15])}...\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Reset\nReset feature engineering to restore original features.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "ds.reset_feature_engineering()\nprint(f\"Features after reset: {len(ds.all_feature_names)}\")"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.11"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}