Hyperparameter Tuning

Hyperparameter optimization strategies.

ModelTunePSO

A class for performing hyperparameter tuning using the PSO algorithm.

__init__(dataset, model)

run(param_bounds: Dict, param_types: Dict=None, dataset: str='train', n_iter: int=10, n_particles: int=10, metric: Union[str, Tuple]=None, n_jobs: int=None, cv=None, error_score=np.nan, random_state: int=0)

Particle Swarm Optimization (PSO) for model tuning.

This method performs hyperparameter optimization using PSO search on the specified model and dataset. It evaluates the model’s performance based on the provided metrics and returns the results in a structured format.

Parameters

param_bounds : dict
Dictionary with parameters names (str) as keys and lists of parameter min and max values (for numerical parameters) or list of parameter categories (for categorical parameters). For example,

{'max_depth': [1, 5],
 'n_estimators': [100, 200],
 'eta': [0.01, 1.0],
 'tree_method': ["exact", "approx", "hist"],
 'centroids': [np.repeat(ds.train_x.min(0).reshape(1, -1), repeats=n_clusters, axis=0).ravel(),
               np.repeat(ds.train_x.max(0).reshape(1, -1), repeats=n_clusters, axis=0).ravel()]
 }

In this setting, parameters can also be an 1D array.

param_types : dict, default=None
Dictionary with parameters names (str) as keys and types of parameter. Available types include “float”, “int”, and “categorical”. For example,

{'max_depth': "int",
 'n_estimators': "int",
 'tree_method': "categorical"}

Note that this is optional, and it may not contain all the keys shown in param_bounds; if a parameter’s type is not specified, then it is treated as float.

dataset : {“main”, “train”, “test”}, default=“train”
The data set for model fitting.

n_iter : int, default=10
Number of iterations of PSO. n_iter trades off runtime vs quality of the solution.

n_particles : int, default=10
Number of particles in each iteration of PSO.

metric : str or tuple, default=None
The performance metric(s). If None,

• For classification: “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”
• For regression: “MSE”, “MAE”, and “R2”

Note that only the first one is used as the optimization objective.

cv : int, cross-validation generator or an iterable, default=None
Determines the cross-validation splitting strategy. Possible inputs for cv are:

• None, to use the default 5-fold cross validation,
• integer, to specify the number of folds in a (Stratified)KFold,
• CV splitter,
• An iterable yielding (train, test) splits as arrays of indices.

n_jobs : int, default=None
Number of jobs to run in parallel. None means 1 unless in a joblib.parallel_backend context. -1 means using all processors. See Glossary for more details.

error_score : ‘raise’ or numeric, default=np.nan
Value to assign to the score if an error occurs in estimator fitting. If set to ‘raise’, the error is raised. If a numeric value is given, FitFailedWarning is raised. This parameter does not affect the refit step, which will always raise the error.

random_state : int, default=0
The random seed for reproducibility.

Returns

ValidationResult
A container object with the following components:

• key: “model_tune_pso”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters

• value: Dictionary containing the optimization history

• table: Tabular format of the optimization history

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

  • “parallel”: Parallel plot of the hyperparameter settings and final performance.
  • ”(<parameter>, <metric>)”: Bar plot showing the performance metric against parameter values.

Examples

ModelTuneOptuna

A class for performing hyperparameter tuning using the optuna Python package.

__init__(dataset, model)

run(param_distributions: Dict, dataset: str='train', n_iter: int=10, sampler: str='tpe', sampler_args: dict=None, metric: Union[str, Tuple]=None, n_jobs: int=None, cv=None, error_score=np.nan, random_state: int=0)

Runs model tuning using Optuna for hyperparameter optimization.

This method performs hyperparameter optimization for the specified model using the Optuna library. It allows users to define parameter distributions, choose a sampling strategy, and specify evaluation metrics. The results of the optimization process are returned in a structured format, including the best parameters and their corresponding scores.

Parameters

param_distributions : dict
A dictionary where keys are parameter names (str) and values are distributions or lists of parameters to try. The distributions must provide a method for sampling (e.g., from scipy.stats), and if a list is provided, it will be sampled uniformly.

dataset : {“main”, “train”, “test”}, default=“train”
Specifies which dataset to use for model fitting. Options include “main” for the entire dataset, “train” for the training subset, and “test” for the testing subset.

n_iter : int, default=10
The number of iterations for the optimization process, controlling the trade-off between runtime and the quality of the solution.

sampler : {“grid”, “random”, “tpe”, “gp”, “cma-es”, “qmc”}, default=“tpe”
The sampling strategy used in optuna.

• “grid” : Grid Search implemented in GridSampler
• “random” : Random Search implemented in RandomSampler
• “tpe” : Tree-structured Parzen Estimator algorithm implemented in TPESampler
• “gp” : Gaussian process-based algorithm implemented in GPSampler
• “cma-es”: CMA-ES based algorithm implemented in CmaEsSampler
• “qmc” : A Quasi Monte Carlo sampling algorithm implemented in QMCSampler

sampler_args : dict, default=None
The arguments passed to the sampler.

dataset : {“main”, “train”, “test”}, default=“train”
The data set for model fitting.

n_iter : int, default=10
Number of iterations of PSO. n_iter trades off runtime vs quality of the solution.

metric : str or tuple, default=None
The performance metric(s). If None,

• For classification: “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”
• For regression: “MSE”, “MAE”, and “R2”

Note that only the first one is used as the optimization objective.

cv : int, cross-validation generator or an iterable, default=None
Determines the cross-validation splitting strategy. Possible inputs for cv are:

• None, to use the default 5-fold cross validation,
• integer, to specify the number of folds in a (Stratified)KFold,
• CV splitter,
• An iterable yielding (train, test) splits as arrays of indices.

n_jobs : int, default=None
The number of jobs to run in parallel. None means 1 unless in a joblib.parallel_backend context; -1 means using all processors.

error_score : ‘raise’ or numeric, default=np.nan
Value to assign to the score if an error occurs in estimator fitting. If set to ‘raise’, the error is raised. If a numeric value is given, FitFailedWarning is raised. This parameter does not affect the refit step, which will always raise the error.

random_state : int, default=0
The seed used for random number generation to ensure reproducibility of results.

Returns

ValidationResult
A container object with the following components:

• key: “model_tune_optuna”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters

• value: Dictionary containing the optimization history

• table: Tabular format of the optimization history

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

  • “parallel”: Parallel plot of the hyperparameter settings and final performance.
  • ”(<parameter>, <metric>)”: Bar plot showing the performance metric against parameter values.

Examples

ModelTuneGridSearch

A class for performing hyperparameter tuning using grid search.

__init__(dataset, model)

run(param_grid: Dict, dataset: str='train', metric: Union[str, Tuple]=None, n_jobs: int=None, cv=None, error_score=np.nan)

Executes a grid search for model tuning.

This method performs hyperparameter optimization using grid search on the specified model and dataset. It evaluates the model’s performance based on the provided metrics and returns the results in a structured format.

Parameters

param_grid : dict
A dictionary where the keys are parameter names (str) and the values are lists of settings to try. Alternatively, it can be a list of such dictionaries to explore multiple grids.

dataset : {“main”, “train”, “test”}, default=“train”
Specifies the dataset to be used for model fitting, with options for main, training, or testing datasets.

metric : str or tuple, default=None
The performance metric(s). If None,

• For classification: “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”
• For regression: “MSE”, “MAE”, and “R2”

Note that only the first one is used as the optimization objective.

cv : int, cross-validation generator or an iterable, default=None
Defines the cross-validation strategy. It can be an integer specifying the number of folds, a CV splitter, or an iterable yielding (train, test) splits as arrays of indices.

n_jobs : int, default=None
The number of jobs to run in parallel. If None, it defaults to 1 unless in a joblib.parallel_backend context. -1 indicates using all processors.

error_score : ‘raise’ or numeric, default=np.nan
Value to assign to the score if an error occurs in estimator fitting. If set to ‘raise’, the error is raised. If a numeric value is given, FitFailedWarning is raised. This parameter does not affect the refit step, which will always raise the error.

Returns

ValidationResult
A container object with the following components:

• key: “model_tune_grid_search”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters

• value: Dictionary containing the optimization history

• table: Tabular format of the optimization history

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

  • “parallel”: Parallel plot of the hyperparameter settings and final performance.
  • ”(<parameter>, <metric>)”: Bar plot showing the performance metric against parameter values.

Examples

ModelTuneRandomSearch

A class for performing hyperparameter tuning using random search.

__init__(dataset, model)

run(param_distributions: Dict, dataset: str='train', n_iter: int=10, metric: Union[str, Tuple]=None, n_jobs: int=None, cv=None, error_score=np.nan, random_state: int=0)

Random Search for model tuning.

This method performs hyperparameter optimization using random search on the specified model and dataset. It evaluates the model’s performance based on the provided metrics and returns the results in a structured format.

Parameters

param_distributions : dict
Dictionary with parameters names (str) as keys and distributions or lists of parameters to try. Distributions must provide a rvs method for sampling (such as those from scipy.stats.distributions). If a list is given, it is sampled uniformly. If a list of dicts is given, first a dict is sampled uniformly, and then a parameter is sampled using that dict as above.

dataset : {“main”, “train”, “test”}, default=“train”
The data set for model fitting.

n_iter : int, default=10
Number of parameter settings that are sampled. n_iter trades off runtime vs quality of the solution.

metric : str or tuple, default=None
The performance metric(s). If None,

• For classification: “ACC”, “AUC”, “F1”, “LogLoss”, “Precision”, “Recall”, and “Brier”
• For regression: “MSE”, “MAE”, and “R2”

Note that only the first one is used as the optimization objective.

cv : int, cross-validation generator or an iterable, default=None
Determines the cross-validation splitting strategy. Possible inputs for cv are:

• None, to use the default 5-fold cross validation,
• integer, to specify the number of folds in a (Stratified)KFold,
• CV splitter,
• An iterable yielding (train, test) splits as arrays of indices.

n_jobs : int, default=None
Number of jobs to run in parallel. None means 1 unless in a joblib.parallel_backend context. -1 means using all processors. See Glossary for more details.

error_score : ‘raise’ or numeric, default=np.nan
Value to assign to the score if an error occurs in estimator fitting. If set to ‘raise’, the error is raised. If a numeric value is given, FitFailedWarning is raised. This parameter does not affect the refit step, which will always raise the error.

random_state : int, default=0
The random seed for reproducibility.

Returns

ValidationResult
A container object with the following components:

• key: “model_tune_random_search”

• data: Name of the dataset used

• model: Name of the model used

• inputs: Input parameters

• value: Dictionary containing the optimization history

• table: Tabular format of the optimization history

• options: Dictionary of visualizations configuration. Run results.plot() to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

  • “parallel”: Parallel plot of the hyperparameter settings and final performance.
  • ”(<parameter>, <metric>)”: Bar plot showing the performance metric against parameter values.

Examples