机器学习模型超参数优化，最频繁使用的5个工具包

news2024/9/23 5:23:20

优化超参数始终是确保模型性能最佳的关键任务。通常，网格搜索、随机搜索和贝叶斯优化等技术是主要使用的方法。

今天分享几个常用于模型超参数优化的 Python 工具包，如下所示：

scikit-learn：使用在指定参数值上进行的网格搜索或随机搜索。
HyperparameterHunter：构建在scikit-learn之上，以使其更易于使用。
Optuna：使用随机搜索、Parzen估计器（TPE）和基于群体的训练。
Hyperopt：使用随机搜索和TPE。
Talos：构建在Keras之上，以使其更易于使用。

现在，让我们看一些使用这些库进行自动编码器模型超参数优化的Python代码示例：

from keras.layers import Input, Dense
from keras.models import Model

# define the Autoencoder
input_layer = Input(shape=(784,))
encoded = Dense(32, activation='relu')(input_layer)
decoded = Dense(784, activation='sigmoid')(encoded)
autoencoder = Model(input_layer, decoded)
autoencoder.compile(optimizer='adam', loss='binary_crossentropy')
autoencoder.fit(X_train, X_train, epochs=100, batch_size=256, validation_data=(X_test, X_test))

scikit-learn

from sklearn.model_selection import GridSearchCV

# define the parameter values that should be searched
param_grid = {'batch_size': [64, 128, 256], 'epochs': [50, 100, 150]}

# create a KFold cross-validator
kfold = KFold(n_splits=10, random_state=7)

# create the grid search object
grid = GridSearchCV(estimator=autoencoder, param_grid=param_grid, cv=kfold)

# fit the grid search object to the training data
grid_result = grid.fit(X_train, X_train)

# print the best parameters and the corresponding score
print(f'Best parameters: {grid_result.best_params_}')
print(f'Best score: {grid_result.best_score_}')

HyperparameterHunter

import HyperparameterHunter as hh

# create a HyperparameterHunter object
hunter = hh.HyperparameterHunter(input_data=X_train, output_data=X_train, model_wrapper=hh.ModelWrapper(autoencoder))

# define the hyperparameter search space
hunter.setup(objective='val_loss', metric='val_loss', optimization_mode='minimize', max_trials=100)
hunter.add_experiment(parameters=hh.Real(0.1, 1, name='learning_rate', digits=3, rounding=4))
hunter.add_experiment(parameters=hh.Real(0.1, 1, name='decay', digits=3, rounding=4))

# perform the hyperparameter search
hunter.hunt(n_jobs=1, gpu_id='0')

# print the best hyperparameters and the corresponding score
print(f'Best hyperparameters: {hunter.best_params}')
print(f'Best score: {hunter.best_score}')

Hyperopt

from hyperopt import fmin, tpe, hp

# define the parameter space
param_space = {'batch_size': hp.quniform('batch_size', 64, 256, 1), 'epochs': hp.quniform('epochs', 50, 150, 1)}

# define the objective function
def objective(params):
    autoencoder.compile(optimizer='adam', loss='binary_crossentropy')
    autoencoder.fit(X_train, X_train, batch_size=params['batch_size'], epochs=params['epochs'], verbose=0)
    scores = autoencoder.evaluate(X_test, X_test, verbose=0)
    return {'loss': scores, 'status': STATUS_OK}

# perform the optimization
best = fmin(objective, param_space, algo=tpe.suggest, max_evals=100)

# print the best parameters and the corresponding score
print(f'Best parameters: {best}')
print(f'Best score: {objective(best)}')

Optuna

import optuna

# define the objective function
def objective(trial):
    batch_size = trial.suggest_int('batch_size', 64, 256)
    epochs = trial.suggest_int('epochs', 50, 150)
    autoencoder.compile(optimizer='adam', loss='binary_crossentropy')
    autoencoder.fit(X_train, X_train, batch_size=batch_size, epochs=epochs, verbose=0)
    score = autoencoder.evaluate(X_test, X_test, verbose=0)
    return score

# create the Optuna study
study = optuna.create_study()

# optimize the hyperparameters
study.optimize(objective, n_trials=100)

# print the best parameters and the corresponding score
print(f'Best parameters: {study.best_params}')
print(f'Best score: {study.best_value}')

Talos

import talos

# define the parameter space
param_space = {'learning_rate': [0.1, 0.01, 0.001], 'decay': [0.1, 0.01, 0.001]}

# define the objective function
def objective(params):
    autoencoder.compile(optimizer='adam', loss='binary_crossentropy', lr=params['learning_rate'], decay=params['decay'])
    history = autoencoder.fit(X_train, X_train, epochs=100, batch_size=256, validation_data=(X_test, X_test), verbose=0)
    score = history.history['val_loss'][-1]
    return score

# perform the optimization
best = talos.Scan(X_train, X_train, params=param_space, model=autoencoder, experiment_name='autoencoder').best_params(objective, n_trials=100)

# print the best parameters and the corresponding score
print(f'Best parameters: {best}')
print(f'Best score: {objective(best)}')