解決:如何設置sparkTrials ?我收到這類型…-磚- 33777

Somi · ‎08-23-2022

我試圖分發hyperparameter tensorflow使用hyperopt調優。keras模型。我用在我fmin sparkTrials:

spark_trials = SparkTrials(並行= 4)

…

best_hyperparam = fmin (fn = CNN_HOF,

空間=空間,

算法= tpe.suggest,

max_evals = tuner_max_evals,

試驗= spark_trials)

但我收到這個錯誤:

_thread TypeError:不能泡菜”。鎖的對象

代碼是工作的唯一方法就是跳過試驗經過評論出線試驗= spark_trials這意味著不會有分布式優化。

知道我如何解決這個問題嗎?

@Tian譚

@Sara Dooley

杜利 · ‎09-01-2022

試試以下:

def CNN_HOF (train_df_pd valid_df_pd、test_df_pd params): # Hyperopt目標函數train_generator = train_data_gen。flow_from_dataframe (dataframe = train_df_pd目錄= images_dir x_col =“文件名”,y_col =目標,target_size = (150、150), class_mode =“分類”,batch_size = train_batch) valid_generator = valid_data_gen。flow_from_dataframe (dataframe = valid_df_pd目錄= images_dir x_col =“文件名”,y_col =目標,target_size = (150、150), class_mode =“分類”,batch_size = valid_batch洗牌= False,種子= 42)test_generator = test_data_gen。flow_from_dataframe (dataframe = test_df_pd目錄= images_dir x_col =“文件名”,y_col =目標,target_size = (150、150), class_mode =“分類”,batch_size = test_batch洗牌= False,種子= 42)mlflow.tensorflow.autolog()模型= model_builder (params dense_size) model.compile (=“categorical_crossentropy”損失,優化器=亞當(),指標=(“準確性”))=曆史模型。適合(train_generator steps_per_epoch = train_step時代= tuner_epochs validation_data = valid_generator validation_steps = valid_step verbose = 2) #評估模型得分=模型。評估(test_generator步驟= 1,verbose = 0) obj_metric[0] =分數返回浮點數(obj_metric)

我假設你dense_size、valid_batch train_batch test_batch image_dir,目標是全局變量。注意,train_df_pd = train_df.toPandas (), valid_df_pd = valid_df.toPandas(),和test_df_pd = test_df.toPandas()在目標函數和熊貓dataframes作為參數。然後你把發電機的目標函數。如果這是,那麼發電機是問題,我們可以做些事情來加速這個過程。

我還想指出,我返回分數作為一個浮動。我想知道分數[0]是一個字典,而不是一個浮點數,但我將它但是你可以打印類型轉化為驗證。

在原帖子查看解決方案

杜利 · ‎08-26-2022

這可能發生當你嚐試序列化keras unserializable層模型。你的模型是什麼樣子?還在搜索空間變量是什麼?你想優化?

Somi · ‎08-26-2022

這是更多的代碼和細節:

空間= {" pool_1”: hp.choice (pool_1, np。論壇(2、5、1、dtype = int)),“conv_1”:惠普。選擇(conv_1, np。不等(16,128,16日dtype = int)),“conv_1b”:惠普。選擇(conv_1b, np。論壇(2、5、1、dtype = int)),“pool_2”: hp.choice (pool_2, np。論壇(2、5、1、dtype = int)),“reg_2”:惠普。選擇(reg_2, np.arange (0.00005, 0.01, 0.00001, dtype =浮動)),“conv_2”:惠普。選擇(“conv_2 np。不等(16,128,16日dtype = int)),“conv_2b”:惠普。選擇(“conv_2b np。論壇(2、5、1、dtype = int)),“pool_3”:惠普。選擇(pool_3, np。論壇(2、5、1、dtype = int)),“reg_3”:惠普。選擇(reg_3, np.arange (0.00005, 0.01, 0.00001, dtype =浮動)),“conv_3”:惠普。選擇(“conv_3 np。不等(16,128,16日dtype = int)),“conv_3b”:惠普。選擇(“conv_3b np。論壇(2、5、1、dtype = int)),“pool_4”:惠普。選擇(pool_4, np。arange(2, 5, 1, dtype=int)), "reg_4" : hp.choice('reg_4', np.arange(0.00005, 0.01, 0.00001, dtype=float)), "conv_4": hp.choice("conv_4", np.arange(16, 128, 16, dtype=int)), "conv_4b": hp.choice("conv_4b",np.arange(2, 5, 1, dtype=int)), "drop_4": hp.choice('drop_4', np.arange(0.00005, 0.01, 0.00001, dtype=float)) }

dense_size def model_builder (params): # CNN builder功能模型=順序()model.add (Conv2D (int (params [' conv_1 ']), (int (params [' conv_1b ']), int (params [' conv_1b '])),激活= relu, input_shape = (150、150、3))) model.add (MaxPooling2D (int (params [' pool_1 ']), int (params [' pool_1 ']))) model.add (Conv2D (int (params [' conv_2 ']), (int (params [' conv_2b ']), int (params [' conv_2b '])),激活= relu, kernel_regularizer = L1L2(浮動(params [' reg_2 ']),浮動(params [' reg_2 '])))) model.add (MaxPooling2D (int (params [' pool_2 ']), int (params [' pool_2 ']))) model.add (Conv2D (int (params [' conv_3 ']), (int (params [' conv_3b ']), int (params [' conv_3b '])),激活= relu, kernel_regularizer = L1L2(浮動(params [' reg_3 ']),浮動(params [' reg_3 '])))) model.add (MaxPooling2D (int (params [' pool_3 ']), int (params [' pool_3 ']))) model.add (Conv2D (int (params [' conv_4 ']), (int (params [' conv_4b ']), int (params [' conv_4b '])),激活= relu, kernel_regularizer = L1L2(浮動(params [' reg_4 ']),浮動(params [' reg_4 '])))) model.add (MaxPooling2D (int (params [' pool_4 ']), int (params [' pool_4 ']))) model.add(輟學(浮動(params [' drop_4 ']))) model.add(平())模型。add(密度(512年,激活= ' relu '))模型。add(密度(dense_size,激活= ' softmax '))回歸模型

def CNN_HOF (params): # Hyperopt目標函數mlflow.tensorflow.autolog()模型= model_builder (params dense_size) model.compile (=“categorical_crossentropy”損失,優化器=亞當(),指標=(“準確性”))=曆史模型。適合(train_generator steps_per_epoch = train_step時代= tuner_epochs validation_data = valid_generator validation_steps = valid_step verbose = 2) #評估模型得分=模型。評估(test_generator步驟= 1,verbose = 0) obj_metric[0] =分數返回{“損失”:obj_metric,“狀態”:STATUS_OK} spark_trials = SparkTrials(並行= 4)…mlflow.start_run (run_name = model_name +“_Tuning”): best_hyperparam = fmin (fn = CNN_HOF空間=空間,算法= tpe。建議,max_evals = tuner_max_evals early_stop_fn = no_progress_loss(10),試驗= spark_trials)

這是完整的錯誤信息:

TypeError回溯(最近調用最後)

模塊在< <命令- 2155252138731800 > >

1如果調優:

- - - - - > 2 Hyperparameter_tuning (model_name)

<命令- 3238776031025884 > Hyperparameter_tuning (model_name)

2與mlflow.start_run (run_name = model_name +“_Tuning”):

3 # mlflow.tensorflow.autolog ()

- - - - - > 4 best_hyperparam = fmin (fn = CNN_HOF,

5 =空間,

6算法= tpe.suggest,

/磚/ .python_edge_libs / hyperopt / fmin。py fmin (fn、空間、藻類、max_evals超時,loss_threshold,試驗,rstate, allow_trials_fmin, pass_expr_memo_ctrl, catch_eval_exceptions,冗長,return_argmin, points_to_evaluate, max_queue_len, show_progressbar, early_stop_fn, trials_save_file)

563年

564如果allow_trials_fmin hasattr(試驗,“fmin”):

- - > 565年返回trials.fmin (

566 fn,

567的空間,

/磚/ .python_edge_libs / hyperopt /儀器。py檢測(kwargs func,自我,args)

25)

26個試題:

推薦- - - - - - > 27 return_val = func (* args, * * kwargs)

28除了exc異常:

與消息29 error_string = " {}: {}“.format(類型(exc)。__name__, str (exc))

/磚/ .python_edge_libs / hyperopt /火花。py fmin(自我,fn、空間、藻類、max_evals超時,loss_threshold, max_queue_len, rstate,冗長,pass_expr_memo_ctrl, catch_eval_exceptions, return_argmin, show_progressbar, early_stop_fn, trials_save_file)

311年除了BaseException e:

312 logger.debug (“fmin線程退出異常了。”)

- - > 313 e

314年:

315 logger.debug(“正常fmin線程退出。”)

/磚/ .python_edge_libs / hyperopt /火花。py fmin(自我,fn、空間、藻類、max_evals超時,loss_threshold, max_queue_len, rstate,冗長,pass_expr_memo_ctrl, catch_eval_exceptions, return_argmin, show_progressbar, early_stop_fn, trials_save_file)

283)

284年

- - > 285 res = fmin (

286 fn,

287的空間,

/磚/ .python_edge_libs / hyperopt / fmin。py fmin (fn、空間、藻類、max_evals超時,loss_threshold,試驗,rstate, allow_trials_fmin, pass_expr_memo_ctrl, catch_eval_exceptions,冗長,return_argmin, points_to_evaluate, max_queue_len, show_progressbar, early_stop_fn, trials_save_file)

592域=基地。域(fn、空間、pass_expr_memo_ctrl = pass_expr_memo_ctrl)

593年

- - > 594 rval = FMinIter (

595算法,

596域,

/磚/ .python_edge_libs / hyperopt / fmin。py __init__(自我、藻類、域、嚐試、rstate,異步的,max_queue_len, poll_interval_secs, max_evals,超時,loss_threshold,冗長,show_progressbar, early_stop_fn, trials_save_file)

180)

181年:

- - > 182 e

183年的試驗。附件(“FMinIter_Domain”) =味精

184年

/磚/ .python_edge_libs / hyperopt / fmin。py __init__(自我、藻類、域、嚐試、rstate,異步的,max_queue_len, poll_interval_secs, max_evals,超時,loss_threshold,冗長,show_progressbar, early_stop_fn, trials_save_file)

163記錄器。警告(“蓋寫老域試驗附件”)

164年試題:

- - > 165年味精= pickler.dumps(域)

166年除了TypeError e:

如果“不能pickle _thread 167。RLock對象”str (e):

/磚/ python / lib / python3.8 /網站/ cloudpickle / cloudpickle_fast。py轉儲(obj、協議、buffer_callback)

71年文件、協議=協議buffer_callback = buffer_callback

72)

磚

如何設置sparkTrials嗎?我收到這個TypeError:不能_thread泡菜”。鎖的對象