Overfitting Analysis (Regression)

This example demonstrates how to analyze model overfitting across different data slices for regression problems using various slicing methods and metrics.

Installation

# To install the required package, use the following command:
# !pip install modeva

Authentication

# To get authentication, use the following command: (To get full access please replace the token to your own token)
# from modeva.utils.authenticate import authenticate
# authenticate(auth_code='eaaa4301-b140-484c-8e93-f9f633c8bacb')

Import required module

from modeva import DataSet
from modeva import TestSuite
from modeva.models import MoLGBMRegressor
from modeva.models import MoXGBRegressor
from modeva.testsuite.utils.slicing_utils import get_data_info

Load and prepare dataset

ds = DataSet()
ds.load(name="BikeSharing")
ds.set_random_split()

ds.scale_numerical(features=("cnt",), method="log1p")
ds.preprocess()

Train models

model1 = MoXGBRegressor()
model1.fit(ds.train_x, ds.train_y)

model2 = MoLGBMRegressor(max_depth=2, verbose=-1, random_state=0)
model2.fit(ds.train_x, ds.train_y.ravel())

MoLGBMRegressor(boosting_type='gbdt', class_weight=None, colsample_bytree=1.0,
                importance_type='split', learning_rate=0.1, max_depth=2,
                min_child_samples=20, min_child_weight=0.001,
                min_split_gain=0.0, n_estimators=100, n_jobs=None,
                num_leaves=31, objective=None, random_state=0, reg_alpha=0.0,
                reg_lambda=0.0, subsample=1.0, subsample_for_bin=200000,
                subsample_freq=0, verbose=-1)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Conduct slicing analysis for overfit regions

ts = TestSuite(ds, model1)
results = ts.diagnose_slicing_overfit(
    train_dataset="train",
    test_dataset="test",
    features="season",
    metric="MAE",
    threshold=0.08
)
results.table

	Feature	Segment	train-Size	train-MAE	test-Size	test-MAE	GAP	Threshold	Weak
0	season	1.0	3419	0.1504	823	0.2434	0.0930	0.08	True
3	season	4.0	3358	0.1474	874	0.2076	0.0602	0.08	False
1	season	2.0	3542	0.1493	867	0.1990	0.0498	0.08	False
2	season	3.0	3584	0.1287	912	0.1684	0.0397	0.08	False

Visualize the results

results.plot()

Analyze data drift between samples above and under the threshold

data_info = get_data_info(res_value=results.value)
data_results = ds.data_drift_test(
    **data_info["season"],
    distance_metric="PSI",
    psi_method="uniform",
    psi_bins=10
)
data_results.plot("summary")

Batch mode 1D slicing analysis

results = ts.diagnose_slicing_overfit(
    train_dataset="train",
    test_dataset="test",
    features=(("hr", ), ("season",), ("temp", )),
    method="auto-xgb1",
    metric="MAE",
)
results.table

	Feature	Segment	train-Size	train-MAE	test-Size	test-MAE	GAP	Threshold	Weak
14	temp	[0.02, 0.24)	1182	0.1497	309	0.2578	0.1081	0.0599	True
10	season	1.0	3419	0.1504	823	0.2434	0.0930	0.0599	True
7	hr	[5.00, 7.00)	1175	0.1780	267	0.2706	0.0926	0.0599	True
15	temp	[0.24, 0.30)	1103	0.1771	279	0.2673	0.0902	0.0599	True
18	temp	[0.44, 0.50)	1098	0.1585	256	0.2469	0.0884	0.0599	True
8	hr	[7.00, 9.00)	1171	0.1490	283	0.2351	0.0862	0.0599	True
6	hr	[3.00, 5.00)	1110	0.2467	284	0.3279	0.0812	0.0599	True
0	hr	[0.00, 3.00)	1737	0.1918	428	0.2692	0.0774	0.0599	True
5	hr	[20.00, 23.00]	2328	0.1258	584	0.1926	0.0669	0.0599	True
2	hr	[14.00, 16.00)	1164	0.1099	294	0.1755	0.0656	0.0599	True
16	temp	[0.30, 0.36)	1514	0.1597	383	0.2237	0.0639	0.0599	True
20	temp	[0.62, 0.68)	1680	0.1493	431	0.2112	0.0619	0.0599	True
13	season	4.0	3358	0.1474	874	0.2076	0.0602	0.0599	True
19	temp	[0.50, 0.62)	2576	0.1507	639	0.2043	0.0536	0.0599	False
11	season	2.0	3542	0.1493	867	0.1990	0.0498	0.0599	False
9	hr	[9.00, 12.00)	1725	0.1178	456	0.1632	0.0454	0.0599	False
17	temp	[0.36, 0.44)	1747	0.1586	458	0.2030	0.0443	0.0599	False
1	hr	[12.00, 14.00)	1166	0.1101	291	0.1508	0.0408	0.0599	False
12	season	3.0	3584	0.1287	912	0.1684	0.0397	0.0599	False
21	temp	[0.68, 0.74)	1303	0.1212	306	0.1581	0.0369	0.0599	False
4	hr	[18.00, 20.00)	1161	0.1131	295	0.1399	0.0268	0.0599	False
22	temp	[0.74, 1.00]	1700	0.0805	415	0.1010	0.0206	0.0599	False
3	hr	[16.00, 18.00)	1166	0.1072	294	0.1263	0.0191	0.0599	False

Batch mode 1D Slicing (all features by setting features=None)

results = ts.diagnose_slicing_overfit(
    train_dataset="train",
    test_dataset="test",
    features=None,
    method="auto-xgb1",
    metric="MAE",
    threshold=0.08
)
results.table

	Feature	Segment	train-Size	train-MAE	test-Size	test-MAE	GAP	Threshold	Weak
52	weathersit	3.0	1123	0.2198	296	0.3900	0.1702	0.08	True
26	hum	[0.88, 1.00]	1638	0.2007	433	0.3311	0.1305	0.08	True
0	atemp	[0.00, 0.24)	1309	0.1437	350	0.2569	0.1132	0.08	True
41	temp	[0.02, 0.24)	1182	0.1497	309	0.2578	0.1081	0.08	True
37	season	1.0	3419	0.1504	823	0.2434	0.0930	0.08	True
...	...	...	...	...	...	...	...	...	...
35	mnth	[8.00, 9.00)	1146	0.1314	329	0.1576	0.0262	0.08	False
6	atemp	[0.70, 1.00]	1438	0.0852	342	0.1073	0.0222	0.08	False
49	temp	[0.74, 1.00]	1700	0.0805	415	0.1010	0.0206	0.08	False
12	hr	[16.00, 18.00)	1166	0.1072	294	0.1263	0.0191	0.08	False
53	weathersit	4.0	3	0.0697	0	NaN	NaN	0.08	False

71 rows × 9 columns

Visualize the results for one feature

results.plot(name="hr", figsize=(6, 6))

Analyze data drift for ‘hr’ feature

data_info = get_data_info(res_value=results.value)
data_results = ds.data_drift_test(
    **data_info["hr"],
    distance_metric="PSI",
    psi_method="uniform",
    psi_bins=10
)
data_results.plot("summary")

Single feature density plot

data_results.plot(("density", "hr"))

2D feature interaction analysis

results = ts.diagnose_slicing_overfit(
    train_dataset="train",
    test_dataset="test",
    features=("hr", "season"),
    method="auto-xgb1",
    metric="MAE",
)
results.table

	Feature1	Segment1	Feature2	Segment2	train-Size	train-MAE	test-Size	test-MAE	GAP	Threshold	Weak
28	hr	[5.00, 7.00)	season	1.0	291	0.2036	58	0.3639	0.1603	0.0599	True
24	hr	[3.00, 5.00)	season	1.0	248	0.2385	62	0.3940	0.1555	0.0599	True
0	hr	[0.00, 3.00)	season	1.0	427	0.1814	99	0.3257	0.1443	0.0599	True
32	hr	[7.00, 9.00)	season	1.0	289	0.1569	69	0.2755	0.1186	0.0599	True
8	hr	[14.00, 16.00)	season	1.0	298	0.1229	62	0.2308	0.1079	0.0599	True
35	hr	[7.00, 9.00)	season	4.0	282	0.1517	70	0.2592	0.1076	0.0599	True
3	hr	[0.00, 3.00)	season	4.0	433	0.1986	95	0.2977	0.0991	0.0599	True
36	hr	[9.00, 12.00)	season	1.0	427	0.1280	110	0.2181	0.0900	0.0599	True
31	hr	[5.00, 7.00)	season	4.0	286	0.1658	66	0.2493	0.0836	0.0599	True
23	hr	[20.00, 23.00]	season	4.0	542	0.1313	166	0.2114	0.0801	0.0599	True
29	hr	[5.00, 7.00)	season	2.0	297	0.1876	70	0.2646	0.0770	0.0599	True
21	hr	[20.00, 23.00]	season	2.0	591	0.1207	145	0.1966	0.0758	0.0599	True
20	hr	[20.00, 23.00]	season	1.0	582	0.1354	138	0.2076	0.0722	0.0599	True
26	hr	[3.00, 5.00)	season	3.0	297	0.2277	76	0.2971	0.0695	0.0599	True
34	hr	[7.00, 9.00)	season	3.0	305	0.1349	71	0.2031	0.0682	0.0599	True
9	hr	[14.00, 16.00)	season	2.0	298	0.1072	70	0.1732	0.0661	0.0599	True
30	hr	[5.00, 7.00)	season	3.0	301	0.1555	73	0.2215	0.0660	0.0599	True
4	hr	[12.00, 14.00)	season	1.0	280	0.1162	80	0.1785	0.0623	0.0599	True
27	hr	[3.00, 5.00)	season	4.0	280	0.2393	69	0.2985	0.0591	0.0599	False
10	hr	[14.00, 16.00)	season	3.0	291	0.1024	85	0.1573	0.0549	0.0599	False
33	hr	[7.00, 9.00)	season	2.0	295	0.1531	73	0.2050	0.0519	0.0599	False
25	hr	[3.00, 5.00)	season	2.0	285	0.2808	77	0.3313	0.0505	0.0599	False
11	hr	[14.00, 16.00)	season	4.0	277	0.1066	77	0.1532	0.0466	0.0599	False
19	hr	[18.00, 20.00)	season	4.0	280	0.1290	74	0.1742	0.0452	0.0599	False
5	hr	[12.00, 14.00)	season	2.0	298	0.1193	70	0.1623	0.0430	0.0599	False
1	hr	[0.00, 3.00)	season	2.0	440	0.2059	112	0.2476	0.0417	0.0599	False
7	hr	[12.00, 14.00)	season	4.0	291	0.1078	62	0.1490	0.0411	0.0599	False
2	hr	[0.00, 3.00)	season	3.0	437	0.1811	122	0.2209	0.0398	0.0599	False
12	hr	[16.00, 18.00)	season	1.0	276	0.1201	86	0.1582	0.0381	0.0599	False
37	hr	[9.00, 12.00)	season	2.0	447	0.1207	105	0.1583	0.0376	0.0599	False
22	hr	[20.00, 23.00]	season	3.0	613	0.1165	135	0.1501	0.0336	0.0599	False
39	hr	[9.00, 12.00)	season	4.0	408	0.1263	120	0.1587	0.0324	0.0599	False
18	hr	[18.00, 20.00)	season	3.0	291	0.0919	83	0.1173	0.0255	0.0599	False
38	hr	[9.00, 12.00)	season	3.0	443	0.0972	121	0.1220	0.0248	0.0599	False
16	hr	[18.00, 20.00)	season	1.0	301	0.1241	59	0.1478	0.0237	0.0599	False
17	hr	[18.00, 20.00)	season	2.0	289	0.1077	79	0.1255	0.0178	0.0599	False
15	hr	[16.00, 18.00)	season	4.0	279	0.1149	75	0.1326	0.0177	0.0599	False
6	hr	[12.00, 14.00)	season	3.0	297	0.0972	79	0.1141	0.0169	0.0599	False
14	hr	[16.00, 18.00)	season	3.0	309	0.0860	67	0.0942	0.0082	0.0599	False
13	hr	[16.00, 18.00)	season	2.0	302	0.1100	66	0.1104	0.0003	0.0599	False

Model comparison

tsc = TestSuite(ds, models=[model1, model2])
results = tsc.compare_slicing_overfit(
    train_dataset="train",
    test_dataset="test",
    features="hr",
    method="quantile",
    bins=10,
    metric="MAE"
)
results.table

	MoXGBRegressor							MoLGBMRegressor
	Feature	Segment	train-Size	train-MAE	test-Size	test-MAE	GAP	Feature	Segment	train-Size	train-MAE	test-Size	test-MAE	GAP
0	hr	[4.00, 7.00)	1727	0.1998	412	0.2903	0.0905	hr	[14.00, 16.00)	1164	0.3669	294	0.4189	0.0520
1	hr	[7.00, 9.00)	1171	0.1490	283	0.2351	0.0862	hr	[7.00, 9.00)	1171	0.7910	283	0.8384	0.0474
2	hr	[0.00, 2.00)	1170	0.1646	280	0.2434	0.0788	hr	[2.00, 4.00)	1125	0.5450	287	0.5694	0.0244
3	hr	[2.00, 4.00)	1125	0.2476	287	0.3234	0.0758	hr	[19.00, 21.00)	1171	0.2197	285	0.2395	0.0197
4	hr	[21.00, 23.00]	1737	0.1268	447	0.1995	0.0727	hr	[21.00, 23.00]	1737	0.2295	447	0.2468	0.0174
5	hr	[14.00, 16.00)	1164	0.1099	294	0.1755	0.0656	hr	[12.00, 14.00)	1166	0.3405	291	0.3426	0.0021
6	hr	[9.00, 12.00)	1725	0.1178	456	0.1632	0.0454	hr	[0.00, 2.00)	1170	0.3658	280	0.3638	-0.0020
7	hr	[12.00, 14.00)	1166	0.1101	291	0.1508	0.0408	hr	[16.00, 19.00)	1747	0.3204	441	0.3074	-0.0130
8	hr	[19.00, 21.00)	1171	0.1199	285	0.1587	0.0387	hr	[9.00, 12.00)	1725	0.3667	456	0.3502	-0.0164
9	hr	[16.00, 19.00)	1747	0.1078	441	0.1282	0.0204	hr	[4.00, 7.00)	1727	0.5684	412	0.5483	-0.0202

Total running time of the script: (0 minutes 4.659 seconds)

Gallery generated by Sphinx-Gallery