Sliced Performance (Regression)

This example demonstrates how to analyze model performance across different data slices for regression problems using various slicing methods and metrics.

Installation

# To install the required package, use the following command:
# !pip install modeva

Authentication

# To get authentication, use the following command: (To get full access please replace the token to your own token)
# from modeva.utils.authenticate import authenticate
# authenticate(auth_code='eaaa4301-b140-484c-8e93-f9f633c8bacb')

Import required modules

from modeva import DataSet
from modeva import TestSuite
from modeva.models import MoLGBMRegressor, MoXGBRegressor
from modeva.testsuite.utils.slicing_utils import get_data_info

Load and prepare dataset

ds = DataSet()
ds.load(name="BikeSharing")
ds.set_target("cnt")
ds.set_random_split()

ds.scale_numerical(features=("cnt",), method="log1p")
ds.preprocess()

Train models

model1 = MoXGBRegressor()
model1.fit(ds.train_x, ds.train_y)

model2 = MoLGBMRegressor(max_depth=2, verbose=-1, random_state=0)
model2.fit(ds.train_x, ds.train_y.ravel())

MoLGBMRegressor(boosting_type='gbdt', class_weight=None, colsample_bytree=1.0,
                importance_type='split', learning_rate=0.1, max_depth=2,
                min_child_samples=20, min_child_weight=0.001,
                min_split_gain=0.0, n_estimators=100, n_jobs=None,
                num_leaves=31, objective=None, random_state=0, reg_alpha=0.0,
                reg_lambda=0.0, subsample=1.0, subsample_for_bin=200000,
                subsample_freq=0, verbose=-1)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Basic slice accuracy analysis

Analyze residual feature importance

ts = TestSuite(ds, model1)

Categorical feature slicing

results = ts.diagnose_slicing_accuracy(features="season", metric="MAE", threshold=0.2)
results.table

# Uniform binning (Numerical feature)
results = ts.diagnose_slicing_accuracy(features="hr", method="uniform", bins=10, metric="MAE")
results.table

	Feature	Segment	Size	MAE	Threshold	Weak
1	hr	[2.30, 4.60)	284	0.3279	0.2037	True
2	hr	[4.60, 6.90)	267	0.2706	0.2037	True
0	hr	[0.00, 2.30)	428	0.2692	0.2037	True
3	hr	[6.90, 9.20)	445	0.2102	0.2037	True
9	hr	[20.70, 23.00]	447	0.1995	0.2037	False
4	hr	[9.20, 11.50)	294	0.1612	0.2037	False
8	hr	[18.40, 20.70)	285	0.1587	0.2037	False
6	hr	[13.80, 16.10)	445	0.1565	0.2037	False
5	hr	[11.50, 13.80)	291	0.1508	0.2037	False
7	hr	[16.10, 18.40)	290	0.1327	0.2037	False

Quantile binning (Numerical feature)

results = ts.diagnose_slicing_accuracy(features="hr", method="quantile", bins=10, metric="MAE")
results.table

	Feature	Segment	Size	MAE	Threshold	Weak
1	hr	[2.00, 4.00)	287	0.3234	0.2037	True
2	hr	[4.00, 7.00)	412	0.2903	0.2037	True
0	hr	[0.00, 2.00)	280	0.2434	0.2037	True
3	hr	[7.00, 9.00)	283	0.2351	0.2037	True
9	hr	[21.00, 23.00]	447	0.1995	0.2037	False
6	hr	[14.00, 16.00)	294	0.1755	0.2037	False
4	hr	[9.00, 12.00)	456	0.1632	0.2037	False
8	hr	[19.00, 21.00)	285	0.1587	0.2037	False
5	hr	[12.00, 14.00)	291	0.1508	0.2037	False
7	hr	[16.00, 19.00)	441	0.1282	0.2037	False

Auto-XGB binning (Numerical feature)

results = ts.diagnose_slicing_accuracy(features="hr", method="auto-xgb1", metric="MAE")
results.table

	Feature	Segment	Size	MAE	Threshold	Weak
1	hr	[3.00, 5.00)	284	0.3279	0.2037	True
2	hr	[5.00, 7.00)	267	0.2706	0.2037	True
0	hr	[0.00, 3.00)	428	0.2692	0.2037	True
3	hr	[7.00, 9.00)	283	0.2351	0.2037	True
9	hr	[20.00, 23.00]	584	0.1926	0.2037	False
6	hr	[14.00, 16.00)	294	0.1755	0.2037	False
4	hr	[9.00, 12.00)	456	0.1632	0.2037	False
5	hr	[12.00, 14.00)	291	0.1508	0.2037	False
8	hr	[18.00, 20.00)	295	0.1399	0.2037	False
7	hr	[16.00, 18.00)	294	0.1263	0.2037	False

Custom binning (Numerical feature)

results = ts.diagnose_slicing_accuracy(
    features="hr",
    method="precompute",
    bins={"hr": (0, 5, 10, 20, 23)},
    metric="MAE"
)
results.table

	Feature	Segment	Size	MAE	Threshold	Weak
0	hr	[0.00, 5.00)	712	0.2926	0.2037	True
1	hr	[5.00, 10.00)	712	0.2329	0.2037	True
3	hr	[20.00, 23.00]	584	0.1926	0.2037	False
2	hr	[10.00, 20.00)	1468	0.1508	0.2037	False

Advanced slicing analysis

# Batch mode 1D slicing
results = ts.diagnose_slicing_accuracy(
    features=(("hr", ), ("season",), ("temp", )),
    method="auto-xgb1",
    metric="MAE"
)
results.plot(name="hr", figsize=(6, 6))

Batch mode 1D Slicing (all features by setting features=None)

results = ts.diagnose_slicing_accuracy(
    features=None,
    method="auto-xgb1",
    metric="MAE"
)
results.table

	Feature	Segment	Size	MAE	Threshold	Weak
0	weathersit	3.0	296	0.3900	0.2037	True
1	hum	[0.88, 1.00]	433	0.3311	0.2037	True
2	hr	[3.00, 5.00)	284	0.3279	0.2037	True
3	hr	[5.00, 7.00)	267	0.2706	0.2037	True
4	hr	[0.00, 3.00)	428	0.2692	0.2037	True
...	...	...	...	...	...	...
67	hr	[18.00, 20.00)	295	0.1399	0.2037	False
68	hr	[16.00, 18.00)	294	0.1263	0.2037	False
69	temp	[0.68, 0.96]	721	0.1252	0.2037	False
70	hum	[0.00, 0.44)	646	0.1181	0.2037	False
71	atemp	[0.68, 0.98]	416	0.1107	0.2037	False

72 rows × 6 columns

2D feature interaction

results = ts.diagnose_slicing_accuracy(
    features=("hr", "season"),
    method="uniform",
    bins=10,
    metric="MAE"
)
results.plot(figsize=(6, 5))

Test distributional difference between weak samples and the rest

data_info = get_data_info(res_value=results.value)
data_results = ds.data_drift_test(**data_info[("hr", "season")],
                                  distance_metric="PSI",
                                  psi_method="uniform",
                                  psi_bins=10)
data_results.plot("summary")

Get the list of available figure names in the result object

data_results.get_figure_names()

['summary', ('density', 'season'), ('density', 'yr'), ('density', 'mnth'), ('density', 'hr'), ('density', 'holiday'), ('density', 'weekday'), ('density', 'workingday'), ('density', 'weathersit'), ('density', 'temp'), ('density', 'atemp'), ('density', 'hum'), ('density', 'windspeed'), ('density', 'cnt')]

Generate a plot in the result object using the figure name

data_results.plot(('density', 'hr'))

Model comparison

# Compare models on numerical feature
tsc = TestSuite(ds, models=[model1, model2])
results = tsc.compare_slicing_accuracy(
    features="hr",
    method="quantile",
    bins=10,
    metric="MAE",
    threshold=0.2
)
results.plot(figsize=(6, 5))

Compare models on categorical feature

results = tsc.compare_slicing_accuracy(
    features="season",
    metric="MAE",
    threshold=None
)
results.table

	MoXGBRegressor				MoLGBMRegressor
	Feature	Segment	Size	MAE	Feature	Segment	Size	MAE
0	season	1.0	823	0.2434	season	1.0	823	0.4486
1	season	4.0	874	0.2076	season	4.0	874	0.4096
2	season	2.0	867	0.1990	season	2.0	867	0.4002
3	season	3.0	912	0.1684	season	3.0	912	0.3850