Residual Analysis (Regression)

Evaluate model residuals.

Installation

# To install the required package, use the following command:
# !pip install modeva

Authentication

# To get authentication, use the following command: (To get full access please replace the token to your own token)
# from modeva.utils.authenticate import authenticate
# authenticate(auth_code='eaaa4301-b140-484c-8e93-f9f633c8bacb')

Import modeva modules

from modeva import DataSet
from modeva import TestSuite
from modeva.models import MoLGBMRegressor

Load BikeSharing Dataset

ds = DataSet()
ds.load(name="BikeSharing")
ds.set_random_split()
ds.set_target("cnt")

ds.scale_numerical(features=("cnt",), method="log1p")
ds.preprocess()

Fit a LGBM model

model = MoLGBMRegressor(name="LGBM-2", max_depth=2, verbose=-1, random_state=0)
model.fit(ds.train_x, ds.train_y.ravel())

MoLGBMRegressor(boosting_type='gbdt', class_weight=None, colsample_bytree=1.0,
                importance_type='split', learning_rate=0.1, max_depth=2,
                min_child_samples=20, min_child_weight=0.001,
                min_split_gain=0.0, n_estimators=100, n_jobs=None,
                num_leaves=31, objective=None, random_state=0, reg_alpha=0.0,
                reg_lambda=0.0, subsample=1.0, subsample_for_bin=200000,
                subsample_freq=0, verbose=-1)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Analyzes residuals feature importance

ts = TestSuite(ds, model)
results = ts.diagnose_residual_interpret(dataset="train")
results.plot()

Visualize the residual against predictor

results = ts.diagnose_residual_analysis(features="hr", dataset="train")
results.plot()

Visualize the residual against response variable

results = ts.diagnose_residual_analysis(features="cnt", dataset="train")
results.plot()

Visualize the residual against model prediction

results = ts.diagnose_residual_analysis(use_prediction=True, dataset="train")
results.plot()

Interpret residual by a XGB depth-2 model

results = ts.diagnose_residual_interpret(dataset='test', n_estimators=100, max_depth=2)

XGB-2 feature performance

results.plot("feature_importance")

XGB-2 effect performance

results.plot("effect_importance")

Further interpretation (main effect plot)

ts_residual = results.value["TestSuite"]
ts_residual.interpret_effects("hr", dataset="test").plot()

Further interpretation (local interpretation)

ts_residual.interpret_local_fi(sample_index=20).plot()

Random forest-based residual clustering analysis (absolute residual)

results = ts.diagnose_residual_cluster(
    dataset="test",
    response_type="abs_residual",
    metric="MAE",
    n_clusters=10,
    cluster_method="rf",
    sample_size=2000,
    n_estimators=100,
    max_depth=5,
)
results.table

	MAE	Size	abs_residual
4	0.6961	349.0	0.6961
6	0.6672	227.0	0.6672
7	0.5565	176.0	0.5565
2	0.5036	240.0	0.5036
1	0.4871	402.0	0.4871
3	0.3387	237.0	0.3387
9	0.2995	648.0	0.2995
8	0.2889	117.0	0.2889
0	0.2883	717.0	0.2883
5	0.2789	363.0	0.2789

Residual value for each cluster

results.plot("cluster_residual")

Performance metric for each cluster

results.plot("cluster_performance")

Feature importance of the random forest model

results.plot("feature_importance")

Analyze data drift for a specific cluster

data_results = ds.data_drift_test(
    **results.value["clusters"][2]["data_info"],
    distance_metric="PSI",
    psi_method="uniform",
    psi_bins=10
)
data_results.plot("summary")

data_results.plot(name=('density', 'hr'))

Random forest-based residual clustering analysis (perturbed residual)

results = ts.diagnose_residual_cluster(
    dataset="test",
    response_type="abs_residual_perturb",
    metric="MAE",
    n_clusters=10,
    cluster_method="rf",
    sample_size=2000,
    n_estimators=100,
    max_depth=5,
)
results.table

	MAE	Size	abs_residual_perturb
7	1.0488	150.0	1.0877
3	0.6053	400.0	0.8061
8	0.6487	120.0	0.7433
1	0.4809	350.0	0.6016
5	0.5565	176.0	0.6006
4	0.3423	242.0	0.4784
2	0.3873	440.0	0.4007
9	0.3002	920.0	0.3294
6	0.2531	277.0	0.2955
0	0.2046	401.0	0.2631

Random forest-based residual clustering analysis (prediction interval width)

results = ts.diagnose_residual_cluster(
    dataset="test",
    response_type="pi_width",
    metric="MAE",
    n_clusters=10,
    cluster_method="rf",
    sample_size=2000,
    n_estimators=100,
    max_depth=5,
)
results.table

	MAE	Size	pi_width
9	0.6680	59.0	1.5924
2	0.4744	179.0	1.5321
5	0.4803	63.0	1.3994
0	0.4606	506.0	1.3455
8	0.3719	98.0	1.3270
4	0.6190	147.0	1.2788
6	0.4209	155.0	1.1828
1	0.3035	188.0	1.1128
7	0.2979	113.0	1.0051
3	0.2265	230.0	0.8901

Compare residuals cluster of multiple models

benchmark = MoLGBMRegressor(name="LGBM-5", max_depth=5, verbose=-1, random_state=0)
benchmark.fit(ds.train_x, ds.train_y.ravel())

tsc = TestSuite(ds, models=[model, benchmark])
results = tsc.compare_residual_cluster(dataset="test")
results.table

	LGBM-2		LGBM-5
	MSE	size	MSE	size
6	16.5739	1.0	9.0096	1.0
8	2.3218	31.0	0.3914	31.0
0	1.6835	64.0	0.4275	64.0
7	0.9895	167.0	0.2554	167.0
4	0.5431	243.0	0.1554	243.0
2	0.4061	420.0	0.1777	420.0
5	0.2458	549.0	0.1153	549.0
1	0.1722	639.0	0.0635	639.0
9	0.0917	650.0	0.0569	650.0
3	0.0405	712.0	0.0321	712.0

results.plot("cluster_performance")