intermediate/ax_multiobjective_nas_tutorial

在 Google Colab 中运行

Colab

下载 Notebook

Notebook

在 GitHub 上查看

GitHub

使用 Ax 进行多目标神经架构搜索

创建日期：2022 年 8 月 19 日 | 最后更新：2024 年 7 月 31 日 | 最后验证：2024 年 11 月 5 日

作者： David Eriksson, Max Balandat 以及 Meta 的自适应实验团队。

在本教程中，我们将展示如何使用 Ax 对流行的 MNIST 数据集上的简单神经网络模型运行多目标神经架构搜索 (NAS)。虽然底层方法通常用于更复杂的模型和更大的数据集，但我们选择了一个可以在笔记本电脑上在 20 分钟内完成端到端运行的教程。

在许多 NAS 应用中，多个目标之间存在天然的权衡。例如，在设备上部署模型时，我们可能希望最大化模型性能（如准确率），同时最大限度地减少功耗、推理延迟或模型大小等相互竞争的指标，以满足部署约束。通常，我们只需牺牲极小的模型性能，就能显著降低计算需求或预测延迟。探索这些权衡的原则性方法是实现可扩展且可持续 AI 的关键，并在 Meta 有许多成功的应用案例 —— 请参阅我们关于自然语言理解模型的案例研究。

在我们的示例中，我们将调整两个隐藏层的宽度、学习率、dropout 概率、批处理大小 (batch size) 和训练 epoch 数量。目标是权衡性能（验证集上的准确率）和模型大小（模型参数数量）。

本教程使用了以下 PyTorch 库：

• PyTorch Lightning（用于指定模型和训练循环）

• TorchX（用于远程/异步运行训练作业）

• BoTorch（为 Ax 算法提供支持的贝叶斯优化库）

定义 TorchX App

我们的目标是优化 mnist_train_nas.py 中定义的 PyTorch Lightning 训练作业。为了使用 TorchX 实现这一点，我们编写了一个辅助函数，它接收训练作业的架构和超参数值，并创建具有适当设置的 TorchX AppDef。

from pathlib import Path

import torchx

from torchx import specs
from torchx.components import utils


def trainer(
    log_path: str,
    hidden_size_1: int,
    hidden_size_2: int,
    learning_rate: float,
    epochs: int,
    dropout: float,
    batch_size: int,
    trial_idx: int = -1,
) -> specs.AppDef:

    # define the log path so we can pass it to the TorchX ``AppDef``
    if trial_idx >= 0:
        log_path = Path(log_path).joinpath(str(trial_idx)).absolute().as_posix()

    return utils.python(
        # command line arguments to the training script
        "--log_path",
        log_path,
        "--hidden_size_1",
        str(hidden_size_1),
        "--hidden_size_2",
        str(hidden_size_2),
        "--learning_rate",
        str(learning_rate),
        "--epochs",
        str(epochs),
        "--dropout",
        str(dropout),
        "--batch_size",
        str(batch_size),
        # other config options
        name="trainer",
        script="mnist_train_nas.py",
        image=torchx.version.TORCHX_IMAGE,
    )

设置 Runner

Ax 的 Runner 抽象允许编写指向各种后端的接口。Ax 已经自带了适用于 TorchX 的 Runner，因此我们只需对其进行配置。在本教程中，我们将以完全异步的方式在本地运行作业。

如果要在集群上启动它们，可以指定不同的 TorchX 调度程序并相应地调整配置。例如，如果你有一个 Kubernetes 集群，只需将调度程序从 local_cwd 更改为 kubernetes 即可。

import tempfile
from ax.runners.torchx import TorchXRunner

# Make a temporary dir to log our results into
log_dir = tempfile.mkdtemp()

ax_runner = TorchXRunner(
    tracker_base="/tmp/",
    component=trainer,
    # NOTE: To launch this job on a cluster instead of locally you can
    # specify a different scheduler and adjust arguments appropriately.
    scheduler="local_cwd",
    component_const_params={"log_path": log_dir},
    cfg={},
)

设置 SearchSpace（搜索空间）

首先，我们定义搜索空间。Ax 同时支持整数和浮点类型的范围参数，以及非数值类型（如字符串）的选择参数。我们将隐藏层大小、学习率、dropout 和 epoch 数量设置为范围参数，并将批处理大小设置为有序选择参数，以强制其为 2 的幂。

from ax.core import (
    ChoiceParameter,
    ParameterType,
    RangeParameter,
    SearchSpace,
)

parameters = [
    # NOTE: In a real-world setting, hidden_size_1 and hidden_size_2
    # should probably be powers of 2, but in our simple example this
    # would mean that ``num_params`` can't take on that many values, which
    # in turn makes the Pareto frontier look pretty weird.
    RangeParameter(
        name="hidden_size_1",
        lower=16,
        upper=128,
        parameter_type=ParameterType.INT,
        log_scale=True,
    ),
    RangeParameter(
        name="hidden_size_2",
        lower=16,
        upper=128,
        parameter_type=ParameterType.INT,
        log_scale=True,
    ),
    RangeParameter(
        name="learning_rate",
        lower=1e-4,
        upper=1e-2,
        parameter_type=ParameterType.FLOAT,
        log_scale=True,
    ),
    RangeParameter(
        name="epochs",
        lower=1,
        upper=4,
        parameter_type=ParameterType.INT,
    ),
    RangeParameter(
        name="dropout",
        lower=0.0,
        upper=0.5,
        parameter_type=ParameterType.FLOAT,
    ),
    ChoiceParameter(  # NOTE: ``ChoiceParameters`` don't require log-scale
        name="batch_size",
        values=[32, 64, 128, 256],
        parameter_type=ParameterType.INT,
        is_ordered=True,
        sort_values=True,
    ),
]

search_space = SearchSpace(
    parameters=parameters,
    # NOTE: In practice, it may make sense to add a constraint
    # hidden_size_2 <= hidden_size_1
    parameter_constraints=[],
)

设置指标 (Metrics)

Ax 有 Metric 的概念，用于定义结果属性以及如何获取这些结果的观测值。这允许例如编码如何从某个分布式执行后端获取数据，并在作为输入传递给 Ax 之前进行预处理。

在本教程中，我们将使用多目标优化，目标是最大化验证准确率并最小化模型参数数量。后者代表模型延迟的一个简单代理，对于小型 ML 模型来说，很难准确估计延迟（在实际应用中，我们会在设备上运行模型时对延迟进行基准测试）。

在我们的示例中，TorchX 将在本地以完全异步的方式运行训练作业，并根据 trial 索引将结果写入 log_dir（参见上面的 trainer() 函数）。我们将定义一个能够识别该日志目录的指标类。通过继承 TensorboardCurveMetric，我们可以直接利用读取和解析 TensorBoard 日志的逻辑。

from ax.metrics.tensorboard import TensorboardMetric
from tensorboard.backend.event_processing import plugin_event_multiplexer as event_multiplexer

class MyTensorboardMetric(TensorboardMetric):

    # NOTE: We need to tell the new TensorBoard metric how to get the id /
    # file handle for the TensorBoard logs from a trial. In this case
    # our convention is to just save a separate file per trial in
    # the prespecified log dir.
    def _get_event_multiplexer_for_trial(self, trial):
        mul = event_multiplexer.EventMultiplexer(max_reload_threads=20)
        mul.AddRunsFromDirectory(Path(log_dir).joinpath(str(trial.index)).as_posix(), None)
        mul.Reload()

        return mul

    # This indicates whether the metric is queryable while the trial is
    # still running. We don't use this in the current tutorial, but Ax
    # utilizes this to implement trial-level early-stopping functionality.
    @classmethod
    def is_available_while_running(cls):
        return False

现在我们可以实例化准确率和模型参数数量的指标。此处 curve_name 是 TensorBoard 日志中指标的名称，而 name 是 Ax 内部使用的指标名称。我们还指定 lower_is_better 来表示这两个指标的期望方向。

val_acc = MyTensorboardMetric(
    name="val_acc",
    tag="val_acc",
    lower_is_better=False,
)
model_num_params = MyTensorboardMetric(
    name="num_params",
    tag="num_params",
    lower_is_better=True,
)

设置 OptimizationConfig

告诉 Ax 应该优化什么的方法是使用 OptimizationConfig。在这里，我们使用 MultiObjectiveOptimizationConfig，因为我们将进行多目标优化。

此外，Ax 支持通过指定目标阈值来对不同指标设置约束，这些阈值界定了我们想要探索的结果空间中的感兴趣区域。在此示例中，我们将验证准确率约束为至少 0.94 (94%)，模型参数数量约束为最多 80,000。

from ax.core import MultiObjective, Objective, ObjectiveThreshold
from ax.core.optimization_config import MultiObjectiveOptimizationConfig


opt_config = MultiObjectiveOptimizationConfig(
    objective=MultiObjective(
        objectives=[
            Objective(metric=val_acc, minimize=False),
            Objective(metric=model_num_params, minimize=True),
        ],
    ),
    objective_thresholds=[
        ObjectiveThreshold(metric=val_acc, bound=0.94, relative=False),
        ObjectiveThreshold(metric=model_num_params, bound=80_000, relative=False),
    ],
)

创建 Ax 实验 (Experiment)

在 Ax 中，Experiment 对象是存储有关问题设置的所有信息的对象。

from ax.core import Experiment

experiment = Experiment(
    name="torchx_mnist",
    search_space=search_space,
    optimization_config=opt_config,
    runner=ax_runner,
)

选择生成策略 (Generation Strategy)

A GenerationStrategy 是我们想要如何执行优化的抽象表示。虽然这可以定制（如果需要，请参阅此教程），但在大多数情况下，Ax 可以根据搜索空间、优化配置和我们想要运行的总试验次数自动确定适当的策略。

通常，Ax 会选择在启动基于模型的贝叶斯优化策略之前评估一定数量的随机配置。

total_trials = 48  # total evaluation budget

from ax.modelbridge.dispatch_utils import choose_generation_strategy

gs = choose_generation_strategy(
    search_space=experiment.search_space,
    optimization_config=experiment.optimization_config,
    num_trials=total_trials,
  )

[INFO 06-03 00:28:33] ax.modelbridge.dispatch_utils: Using Models.BOTORCH_MODULAR since there is at least one ordered parameter and there are no unordered categorical parameters.
[INFO 06-03 00:28:33] ax.modelbridge.dispatch_utils: Calculating the number of remaining initialization trials based on num_initialization_trials=None max_initialization_trials=None num_tunable_parameters=6 num_trials=48 use_batch_trials=False
[INFO 06-03 00:28:33] ax.modelbridge.dispatch_utils: calculated num_initialization_trials=9
[INFO 06-03 00:28:33] ax.modelbridge.dispatch_utils: num_completed_initialization_trials=0 num_remaining_initialization_trials=9
[INFO 06-03 00:28:33] ax.modelbridge.dispatch_utils: `verbose`, `disable_progbar`, and `jit_compile` are not yet supported when using `choose_generation_strategy` with ModularBoTorchModel, dropping these arguments.
[INFO 06-03 00:28:33] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+BoTorch', steps=[Sobol for 9 trials, BoTorch for subsequent trials]). Iterations after 9 will take longer to generate due to model-fitting.

配置调度程序 (Scheduler)

The Scheduler 充当优化的循环控制器。它与后端通信以启动试验、检查其状态并检索结果。在本教程中，它只是读取和解析本地保存的日志。在远程执行设置中，它将调用 API。Ax Scheduler 教程中的以下插图总结了 Scheduler 如何与用于运行试验评估的外部系统交互。

The Scheduler 需要 Experiment 和 GenerationStrategy。可以通过 SchedulerOptions 传递一组选项。在这里，我们配置了总评估次数以及 max_pending_trials（应同时运行的最大试验次数）。在我们的本地设置中，这是作为独立进程运行的训练作业数量；而在远程执行设置中，这将是您想要并行使用的机器数量。

from ax.service.scheduler import Scheduler, SchedulerOptions

scheduler = Scheduler(
    experiment=experiment,
    generation_strategy=gs,
    options=SchedulerOptions(
        total_trials=total_trials, max_pending_trials=4
    ),
)

[WARNING 06-03 00:28:33] ax.service.utils.with_db_settings_base: Ax currently requires a sqlalchemy version below 2.0. This will be addressed in a future release. Disabling SQL storage in Ax for now, if you would like to use SQL storage please install Ax with mysql extras via `pip install ax-platform[mysql]`.
[INFO 06-03 00:28:33] Scheduler: `Scheduler` requires experiment to have immutable search space and optimization config. Setting property immutable_search_space_and_opt_config to `True` on experiment.

运行优化

现在一切配置就绪，我们可以让 Ax 以完全自动化的方式运行优化。调度程序将定期检查日志以获取所有当前正在运行的试验的状态，如果试验完成，调度程序将更新其在实验中的状态，并获取贝叶斯优化算法所需的观测值。

scheduler.run_all_trials()

[INFO 06-03 00:28:33] Scheduler: Fetching data for newly completed trials: [].
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning: Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.
  warn("Encountered exception in computing model fit quality: " + str(e))
[INFO 06-03 00:28:33] Scheduler: Running trials [0]...
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning: Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.
  warn("Encountered exception in computing model fit quality: " + str(e))
[INFO 06-03 00:28:33] Scheduler: Running trials [1]...
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning: Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.
  warn("Encountered exception in computing model fit quality: " + str(e))
[INFO 06-03 00:28:34] Scheduler: Running trials [2]...
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning: Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.
  warn("Encountered exception in computing model fit quality: " + str(e))
[INFO 06-03 00:28:35] Scheduler: Running trials [3]...
[INFO 06-03 00:28:36] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:28:36] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 06-03 00:28:37] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:28:37] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 06-03 00:28:38] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:28:38] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 06-03 00:28:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:28:41] Scheduler: Retrieved FAILED trials: [0].
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning: Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.
  warn("Encountered exception in computing model fit quality: " + str(e))
[INFO 06-03 00:28:41] Scheduler: Running trials [4]...
[INFO 06-03 00:28:42] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:28:42] Scheduler: Retrieved FAILED trials: [1].
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning: Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.
  warn("Encountered exception in computing model fit quality: " + str(e))
[INFO 06-03 00:28:42] Scheduler: Running trials [5]...
[INFO 06-03 00:28:43] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:28:43] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 06-03 00:28:44] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:28:44] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 06-03 00:28:45] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:28:45] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 06-03 00:28:47] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:28:47] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 06-03 00:28:51] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:28:51] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 4).
[INFO 06-03 00:28:56] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:28:56] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 4).
[INFO 06-03 00:29:03] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:29:03] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 4).
[INFO 06-03 00:29:15] Scheduler: Fetching data for newly completed trials: [2, 5].
[INFO 06-03 00:29:15] Scheduler: Retrieved COMPLETED trials: [2, 5].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning: Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.
  warn("Encountered exception in computing model fit quality: " + str(e))
[INFO 06-03 00:29:15] Scheduler: Running trials [6]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning: Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.
  warn("Encountered exception in computing model fit quality: " + str(e))
[INFO 06-03 00:29:16] Scheduler: Running trials [7]...
[INFO 06-03 00:29:17] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:29:17] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 06-03 00:29:18] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:29:18] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 06-03 00:29:19] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:29:19] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 06-03 00:29:22] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:29:22] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 06-03 00:29:25] Scheduler: Fetching data for newly completed trials: [3].
[INFO 06-03 00:29:25] Scheduler: Retrieved COMPLETED trials: [3].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning: Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.
  warn("Encountered exception in computing model fit quality: " + str(e))
[INFO 06-03 00:29:25] Scheduler: Running trials [8]...
[INFO 06-03 00:29:26] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:29:26] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 06-03 00:29:27] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:29:27] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 06-03 00:29:29] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:29:29] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 06-03 00:29:31] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:29:31] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 06-03 00:29:34] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:29:34] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 4).
[INFO 06-03 00:29:39] Scheduler: Fetching data for newly completed trials: [7].
[INFO 06-03 00:29:39] Scheduler: Retrieved COMPLETED trials: [7].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning: Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.
  warn("Encountered exception in computing model fit quality: " + str(e))
[INFO 06-03 00:29:39] Scheduler: Running trials [9]...
[INFO 06-03 00:29:39] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:29:39] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 06-03 00:29:40] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:29:40] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 06-03 00:29:42] Scheduler: Fetching data for newly completed trials: [4].
[INFO 06-03 00:29:42] Scheduler: Retrieved COMPLETED trials: [4].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning: Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.
  warn("Encountered exception in computing model fit quality: " + str(e))
[INFO 06-03 00:29:42] Scheduler: Running trials [10]...
[INFO 06-03 00:29:43] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:29:43] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 06-03 00:29:44] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:29:44] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 06-03 00:29:45] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:29:45] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 06-03 00:29:48] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:29:48] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 06-03 00:29:51] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:29:51] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 4).
[INFO 06-03 00:29:56] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:29:56] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 4).
[INFO 06-03 00:30:04] Scheduler: Fetching data for newly completed trials: [8, 10].
[INFO 06-03 00:30:04] Scheduler: Retrieved COMPLETED trials: [8, 10].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:30:10] Scheduler: Running trials [11]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:30:17] Scheduler: Running trials [12]...
[INFO 06-03 00:30:17] Scheduler: Fetching data for newly completed trials: [6].
[INFO 06-03 00:30:17] Scheduler: Retrieved COMPLETED trials: [6].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:30:22] Scheduler: Running trials [13]...
[INFO 06-03 00:30:23] Scheduler: Fetching data for newly completed trials: [9].
[INFO 06-03 00:30:23] Scheduler: Retrieved COMPLETED trials: [9].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:30:24] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:30:24] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 06-03 00:30:25] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:30:25] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 06-03 00:30:27] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:30:27] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 06-03 00:30:29] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:30:29] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 06-03 00:30:32] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:30:32] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 06-03 00:30:37] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:30:37] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 06-03 00:30:45] Scheduler: Fetching data for newly completed trials: [11].
[INFO 06-03 00:30:45] Scheduler: Retrieved COMPLETED trials: [11].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:30:49] Scheduler: Running trials [14]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:30:52] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:30:52] Scheduler: Fetching data for newly completed trials: [12].
[INFO 06-03 00:30:52] Scheduler: Retrieved COMPLETED trials: [12].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:30:58] Scheduler: Running trials [15]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:31:01] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:31:01] Scheduler: Fetching data for newly completed trials: [13].
[INFO 06-03 00:31:01] Scheduler: Retrieved COMPLETED trials: [13].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:31:09] Scheduler: Running trials [16]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:31:13] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:31:13] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:31:13] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 06-03 00:31:14] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:31:14] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 06-03 00:31:15] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:31:15] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 06-03 00:31:18] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:31:18] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 06-03 00:31:21] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:31:21] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 06-03 00:31:26] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:31:26] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 06-03 00:31:34] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:31:34] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 06-03 00:31:45] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:31:45] Scheduler: Waiting for completed trials (for 17 sec, currently running trials: 3).
[INFO 06-03 00:32:02] Scheduler: Fetching data for newly completed trials: 14 - 15.
[INFO 06-03 00:32:02] Scheduler: Retrieved COMPLETED trials: 14 - 15.
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:32:09] Scheduler: Running trials [17]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:32:18] Scheduler: Running trials [18]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:32:23] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:32:23] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:32:23] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 06-03 00:32:24] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:32:24] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 06-03 00:32:25] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:32:25] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 06-03 00:32:28] Scheduler: Fetching data for newly completed trials: [16].
[INFO 06-03 00:32:28] Scheduler: Retrieved COMPLETED trials: [16].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:32:34] Scheduler: Running trials [19]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:32:38] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:32:38] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:32:38] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 06-03 00:32:39] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:32:39] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 06-03 00:32:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:32:41] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 06-03 00:32:43] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:32:43] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 06-03 00:32:47] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:32:47] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 06-03 00:32:52] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:32:52] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 06-03 00:32:59] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:32:59] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 06-03 00:33:11] Scheduler: Fetching data for newly completed trials: [18].
[INFO 06-03 00:33:11] Scheduler: Retrieved COMPLETED trials: [18].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:33:20] Scheduler: Running trials [20]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:33:25] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:33:25] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:33:25] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 06-03 00:33:26] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:33:26] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 06-03 00:33:27] Scheduler: Fetching data for newly completed trials: [19].
[INFO 06-03 00:33:27] Scheduler: Retrieved COMPLETED trials: [19].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:33:36] Scheduler: Running trials [21]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:33:42] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:33:42] Scheduler: Fetching data for newly completed trials: [17].
[INFO 06-03 00:33:42] Scheduler: Retrieved COMPLETED trials: [17].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:33:52] Scheduler: Running trials [22]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:33:59] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:33:59] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:33:59] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 06-03 00:34:00] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:34:00] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 06-03 00:34:02] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:34:02] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 06-03 00:34:04] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:34:04] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 06-03 00:34:07] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:34:07] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 06-03 00:34:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:34:12] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 06-03 00:34:20] Scheduler: Fetching data for newly completed trials: [20].
[INFO 06-03 00:34:20] Scheduler: Retrieved COMPLETED trials: [20].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:34:29] Scheduler: Running trials [23]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:34:35] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:34:35] Scheduler: Fetching data for newly completed trials: [21].
[INFO 06-03 00:34:35] Scheduler: Retrieved COMPLETED trials: [21].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:34:46] Scheduler: Running trials [24]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:34:54] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:34:54] Scheduler: Fetching data for newly completed trials: [22].
[INFO 06-03 00:34:54] Scheduler: Retrieved COMPLETED trials: [22].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:35:09] Scheduler: Running trials [25]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:35:19] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:35:19] Scheduler: Fetching data for newly completed trials: [23].
[INFO 06-03 00:35:19] Scheduler: Retrieved COMPLETED trials: [23].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:35:28] Scheduler: Running trials [26]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:35:38] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:35:38] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:35:38] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 06-03 00:35:39] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:35:39] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 06-03 00:35:40] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:35:40] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 06-03 00:35:42] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:35:42] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 06-03 00:35:46] Scheduler: Fetching data for newly completed trials: [24].
[INFO 06-03 00:35:46] Scheduler: Retrieved COMPLETED trials: [24].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:35:54] Scheduler: Running trials [27]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:35:58] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:35:59] Scheduler: Fetching data for newly completed trials: [25].
[INFO 06-03 00:35:59] Scheduler: Retrieved COMPLETED trials: [25].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:36:07] Scheduler: Running trials [28]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:36:13] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:36:13] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:36:13] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 06-03 00:36:14] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:36:14] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 06-03 00:36:15] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:36:15] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 06-03 00:36:18] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:36:18] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 06-03 00:36:21] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:36:21] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 06-03 00:36:26] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:36:26] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 06-03 00:36:34] Scheduler: Fetching data for newly completed trials: [26].
[INFO 06-03 00:36:34] Scheduler: Retrieved COMPLETED trials: [26].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:36:44] Scheduler: Running trials [29]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:36:52] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:36:52] Scheduler: Fetching data for newly completed trials: [27].
[INFO 06-03 00:36:52] Scheduler: Retrieved COMPLETED trials: [27].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:37:01] Scheduler: Running trials [30]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:37:05] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:37:05] Scheduler: Fetching data for newly completed trials: [28].
[INFO 06-03 00:37:05] Scheduler: Retrieved COMPLETED trials: [28].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:37:19] Scheduler: Running trials [31]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:37:28] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:37:28] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:37:28] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 06-03 00:37:29] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:37:29] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 06-03 00:37:31] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:37:31] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 06-03 00:37:33] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:37:33] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 06-03 00:37:36] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:37:36] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 06-03 00:37:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:37:41] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 06-03 00:37:49] Scheduler: Fetching data for newly completed trials: [29].
[INFO 06-03 00:37:49] Scheduler: Retrieved COMPLETED trials: [29].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:38:00] Scheduler: Running trials [32]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:38:06] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:38:06] Scheduler: Fetching data for newly completed trials: [30].
[INFO 06-03 00:38:06] Scheduler: Retrieved COMPLETED trials: [30].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:38:22] Scheduler: Running trials [33]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:38:33] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:38:33] Scheduler: Fetching data for newly completed trials: [31].
[INFO 06-03 00:38:33] Scheduler: Retrieved COMPLETED trials: [31].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:38:46] Scheduler: Running trials [34]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:38:51] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:38:51] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:38:51] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 06-03 00:38:52] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:38:52] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 06-03 00:38:54] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:38:54] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 06-03 00:38:56] Scheduler: Fetching data for newly completed trials: [32].
[INFO 06-03 00:38:56] Scheduler: Retrieved COMPLETED trials: [32].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:39:04] Scheduler: Running trials [35]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:39:09] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:39:09] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:39:09] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 06-03 00:39:10] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:39:10] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 06-03 00:39:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:39:12] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 06-03 00:39:14] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:39:14] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 06-03 00:39:18] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:39:18] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 06-03 00:39:23] Scheduler: Fetching data for newly completed trials: [33].
[INFO 06-03 00:39:23] Scheduler: Retrieved COMPLETED trials: [33].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:39:32] Scheduler: Running trials [36]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:39:36] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:39:36] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:39:36] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 06-03 00:39:37] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:39:37] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 06-03 00:39:39] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:39:39] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 06-03 00:39:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:39:41] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 06-03 00:39:44] Scheduler: Fetching data for newly completed trials: [34].
[INFO 06-03 00:39:44] Scheduler: Retrieved COMPLETED trials: [34].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:40:01] Scheduler: Running trials [37]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:40:09] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:40:09] Scheduler: Fetching data for newly completed trials: [35].
[INFO 06-03 00:40:09] Scheduler: Retrieved COMPLETED trials: [35].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:40:22] Scheduler: Running trials [38]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:40:31] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:40:31] Scheduler: Fetching data for newly completed trials: [36].
[INFO 06-03 00:40:31] Scheduler: Retrieved COMPLETED trials: [36].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:40:40] Scheduler: Running trials [39]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:40:46] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:40:46] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:40:46] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 06-03 00:40:47] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:40:47] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 06-03 00:40:49] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:40:49] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 06-03 00:40:51] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:40:51] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 06-03 00:40:54] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:40:54] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 06-03 00:40:59] Scheduler: Fetching data for newly completed trials: [37].
[INFO 06-03 00:40:59] Scheduler: Retrieved COMPLETED trials: [37].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:41:12] Scheduler: Running trials [40]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:41:22] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:41:22] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:41:22] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 06-03 00:41:23] Scheduler: Fetching data for newly completed trials: [38].
[INFO 06-03 00:41:23] Scheduler: Retrieved COMPLETED trials: [38].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:41:36] Scheduler: Running trials [41]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:41:45] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:41:45] Scheduler: Fetching data for newly completed trials: [39].
[INFO 06-03 00:41:45] Scheduler: Retrieved COMPLETED trials: [39].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:41:56] Scheduler: Running trials [42]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:42:03] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:42:03] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:42:03] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 06-03 00:42:04] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:42:04] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 06-03 00:42:05] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:42:05] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 06-03 00:42:07] Scheduler: Fetching data for newly completed trials: [40].
[INFO 06-03 00:42:07] Scheduler: Retrieved COMPLETED trials: [40].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:42:19] Scheduler: Running trials [43]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:42:27] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:42:27] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:42:27] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 06-03 00:42:28] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:42:28] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 06-03 00:42:29] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:42:29] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 06-03 00:42:31] Scheduler: Fetching data for newly completed trials: [41].
[INFO 06-03 00:42:31] Scheduler: Retrieved COMPLETED trials: [41].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:42:48] Scheduler: Running trials [44]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:42:57] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:42:57] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:42:57] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 06-03 00:42:58] Scheduler: Fetching data for newly completed trials: [42].
[INFO 06-03 00:42:58] Scheduler: Retrieved COMPLETED trials: [42].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:43:11] Scheduler: Running trials [45]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:43:22] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:43:22] Scheduler: Fetching data for newly completed trials: [43].
[INFO 06-03 00:43:22] Scheduler: Retrieved COMPLETED trials: [43].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:43:33] Scheduler: Running trials [46]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:43:42] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 06-03 00:43:42] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:43:42] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 06-03 00:43:43] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:43:43] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 06-03 00:43:44] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:43:44] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 06-03 00:43:47] Scheduler: Fetching data for newly completed trials: [44].
[INFO 06-03 00:43:47] Scheduler: Retrieved COMPLETED trials: [44].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:44:04] Scheduler: Running trials [47]...
[INFO 06-03 00:44:05] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:44:05] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 06-03 00:44:06] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:44:06] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 06-03 00:44:07] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:44:07] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 06-03 00:44:10] Scheduler: Fetching data for newly completed trials: [45].
[INFO 06-03 00:44:10] Scheduler: Retrieved COMPLETED trials: [45].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:44:10] Scheduler: Done submitting trials, waiting for remaining 2 running trials...
[INFO 06-03 00:44:10] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:44:10] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 2).
[INFO 06-03 00:44:11] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:44:11] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 2).
[INFO 06-03 00:44:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:44:12] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 2).
[INFO 06-03 00:44:14] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:44:14] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 2).
[INFO 06-03 00:44:18] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:44:18] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 2).
[INFO 06-03 00:44:23] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:44:23] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 2).
[INFO 06-03 00:44:31] Scheduler: Fetching data for newly completed trials: [46].
[INFO 06-03 00:44:31] Scheduler: Retrieved COMPLETED trials: [46].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[INFO 06-03 00:44:31] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:44:31] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 1).
[INFO 06-03 00:44:32] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:44:32] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 1).
[INFO 06-03 00:44:33] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:44:33] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 1).
[INFO 06-03 00:44:35] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:44:35] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 1).
[INFO 06-03 00:44:39] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:44:39] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 1).
[INFO 06-03 00:44:44] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:44:44] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 1).
[INFO 06-03 00:44:51] Scheduler: Fetching data for newly completed trials: [].
[INFO 06-03 00:44:51] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 1).
[INFO 06-03 00:45:03] Scheduler: Fetching data for newly completed trials: [47].
[INFO 06-03 00:45:03] Scheduler: Retrieved COMPLETED trials: [47].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(

OptimizationResult()

评估结果

我们现在可以使用 Ax 包含的辅助函数和可视化工具来检查优化的结果。

首先，我们生成一个数据框，其中包含实验结果的摘要。此数据框中的每一行对应一个试验（即已运行的训练作业），并包含有关试验状态、已评估的参数配置以及观测到的指标值的信息。这提供了一种对优化进行合理性检查的简便方法。

from ax.service.utils.report_utils import exp_to_df

df = exp_to_df(experiment)
df.head(10)

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[WARNING 06-03 00:45:03] ax.service.utils.report_utils: Column reason missing for all trials. Not appending column.

	trial_index（试验索引）	arm_name	trial_status（试验状态）	generation_method（生成方法）	is_feasible（是否可行）	参数数量	val_acc（验证准确率）	hidden_size_1	hidden_size_2	learning_rate（学习率）	epochs（轮数）	dropout	batch_size（批处理大小）
0	0	0_0	FAILED（失败）	Sobol（索博尔序列）	NaN	NaN	NaN	56	57	0.001477	1	0.444781	32
1	1	1_0	FAILED（失败）	Sobol（索博尔序列）	NaN	NaN	NaN	16	40	0.000167	4	0.241499	128
2	2	2_0	COMPLETED（已完成）	Sobol（索博尔序列）	真	32024.0	0.951565	34	122	0.003463	2	0.114314	256
3	3	3_0	COMPLETED（已完成）	Sobol（索博尔序列）	真	64430.0	0.943367	80	19	0.000398	3	0.317570	64
4	4	4_0	COMPLETED（已完成）	Sobol（索博尔序列）	假	100164.0	0.955309	114	87	0.000218	3	0.289093	32
5	5	5_0	COMPLETED（已完成）	Sobol（索博尔序列）	真	30230.0	0.952619	37	26	0.001876	2	0.023651	128
6	6	6_0	COMPLETED（已完成）	Sobol（索博尔序列）	真	21196.0	0.942334	24	70	0.000924	4	0.150870	256
7	7	7_0	COMPLETED（已完成）	Sobol（索博尔序列）	假	54252.0	0.912146	66	33	0.007825	1	0.416269	64
8	8	8_0	COMPLETED（已完成）	Sobol（索博尔序列）	真	63360.0	0.956871	70	106	0.000740	2	0.201245	64
9	9	9_0	COMPLETED（已完成）	Sobol（索博尔序列）	假	18560.0	0.893894	23	16	0.006321	3	0.498218	256

我们还可以可视化验证准确率与模型参数数量之间的帕累托前沿 (Pareto frontier) 权衡。

提示

Ax 使用 Plotly 生成交互式图表，允许您执行缩放、裁剪或悬停等操作，以便查看绘图组件的详细信息。尝试一下，如果您想了解更多信息，请查看可视化教程。

最终的优化结果如下图所示，其中颜色对应于每个试验的迭代次数。我们看到，我们的方法成功探索了权衡，并发现了验证准确率高的大模型以及验证准确率相对较低的小模型。

from ax.service.utils.report_utils import _pareto_frontier_scatter_2d_plotly

_pareto_frontier_scatter_2d_plotly(experiment)

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat(
[WARNING 06-03 00:45:03] ax.service.utils.report_utils: Column reason missing for all trials. Not appending column.

为了更好地理解我们的代理模型对黑盒目标了解了什么，我们可以查看留一法交叉验证 (leave-one-out cross validation) 的结果。由于我们的模型是高斯过程，它们不仅提供点预测，还提供关于这些预测的不确定性估计。一个好的模型意味着预测均值（图中的点）接近 45 度线，并且置信区间以预期的频率覆盖 45 度线（这里我们使用 95% 的置信区间，因此我们预计它们在 95% 的时间内包含真实观测值）。

如下面的图所示，模型大小 (num_params) 指标比验证准确率 (val_acc) 指标更容易建模。

from ax.modelbridge.cross_validation import compute_diagnostics, cross_validate
from ax.plot.diagnostic import interact_cross_validation_plotly
from ax.utils.notebook.plotting import init_notebook_plotting, render

cv = cross_validate(model=gs.model)  # The surrogate model is stored on the ``GenerationStrategy``
compute_diagnostics(cv)

interact_cross_validation_plotly(cv)

我们还可以制作等高线图，以更好地理解不同目标如何依赖于两个输入参数。在下图中，我们显示了模型预测的验证准确率随两个隐藏层大小的变化情况。随着隐藏层大小的增加，验证准确率明显提高。

from ax.plot.contour import interact_contour_plotly

interact_contour_plotly(model=gs.model, metric_name="val_acc")

同样，我们在下图中显示了模型参数数量随隐藏层大小的变化，并看到它也随着隐藏层大小的增加而增加（对 hidden_size_1 的依赖性要大得多）。

interact_contour_plotly(model=gs.model, metric_name="num_params")