Evaluate Experiments¶
Once you start modeling, Workbench begins to construct your model Leaderboard. The Leaderboard is a ranked list of models that provides a summary of information, including scoring information, for each model built in an experiment.
Once you start modeling, Workbench begins to construct your model Leaderboard, a list of models ranked by performance, to help with quick model evaluation. The Leaderboard provides a summary of information, including scoring information, for each model built in an experiment. From the Leaderboard, you can click a model to access visualizations for further exploration. Using these tools can help to assess what to do in your next experiment.
After Workbench completes Quick mode on the 64% sample size phase, the most accurate model is selected and trained on 100% of the data. That model is marked with the Prepared for Deployment badge.
Manage the Leaderboard¶
There are several controls available, described in the next sections, for navigating the Leaderboard.
エクスペリメント情報を表示¶
Click View experiment info to view a summary of information about the experiment. These are the parameters used to build the models on this Leaderboard.
| フィールド | Reports... |
|---|---|
| 作成完了 | A time stamp indicating the creation of the experiment's Leaderboard as well as the user who initiated the model run. |
| データセット | The name, number of features, and number of rows in the modeling dataset. This is the same information available from the data preview page. |
| ターゲット | The feature selected as the basis for predictions, the resulting project type, and the optimization metric used to define how to score the experiment's models. You can change the metric the Leaderboard is sorted by, but the metric displayed in the summary is the one used for the build. |
| パーティション | Details of the partitioning done for the experiment, either the default or modified. |
モデルを絞り込む¶
Filtering makes viewing and focusing on relevant models easier. Click Filter models to set the criteria for the models that Workbench displays on the Leaderboard. The choices available for each filter are dependent on the experiment and/or model type—they were used in at least one Leaderboard model—and will potentially change as models are added to the experiment.
| フィルター | Displays models that... |
|---|---|
| ラベルが設定されたモデル | Have been assigned the listed tag, either starred models or models recommended for deployment. |
| 特徴量セット | Were built with the selected feature list. |
| サンプルサイズ | Were trained on the selected sample size. |
| モデルファミリー | Are part of the selected model family:
|
モデルの並べ替え条件¶
By default, the Leaderboard sorts models based on the score of the validation partition, using the selected optimization metric. You can, however, use the Sort models by control to change the basis of the display parameter when evaluating models.
Note that although Workbench built the project using the most appropriate metric for your data, it computes many applicable metrics on each of the models. 構築が完了した後、別の指標に基づいてリーダーボードのリストを再表示できます。 モデル内の値は変更されませんが、この代替指標でのパフォーマンスに基づいてモデルの一覧の表示順序が変更されます。
See the page on optimization metrics for detailed information on each.
Controls¶
Workbench provides simple, quick shorthand controls:
| アイコン | アクション |
|---|---|
 |
Reruns Quick mode with a different feature list If you select a feature list that has already been run, Workbench will replace and deleted models or make no changes. |
 |
Duplicates the experiment, with an option to reuse just the dataset, or the dataset and settings. |
 |
Deletes the experiment and its models. If the experiment is being used by an application, you cannot delete it. |
 |
Slides the Leaderboard panel closed to make additional room for, for example, viewing insights. |
インサイト¶
Model insights help to interpret, explain, and validate what drives a model’s predictions. Available insights are dependent on experiment type, but may include:
To see a model's insights, click on the model in the left-pane Leaderboard.
特徴量のインパクト¶
特徴量のインパクト モデルの決定を最も強力に推進している特徴量の高レベルの視覚化を提供します。 It is available for all model types and is an on-demand feature, meaning that for all but models prepared for deployment, you must initiate a calculation to see the results.
ROC曲線¶
For classification experiments, the ROC Curve tab provides the following tools for exploring classification, performance, and statistics related to a selected model at any point on the probability scale:
残差¶
For regression experiments, the Residuals tab helps to clearly understand a model's predictive performance and validity. このタブでは、使用するデータセットの実測値に相対的にモデルがどのように正比例的にスケールするかを測定できます。 It provides multiple scatter plots and a histogram to assist your residual analysis:
- 予測値と実測値の比較
- 残差と実測値の比較
- 残差と予測値の比較
- 残差ヒストグラム
Train on new settings¶
Once the Leaderboard is populated, you can retrain an existing model to create a new Leaderboard model. モデルを再トレーニングするには：
- Select a model from the Leaderboard by clicking on it.
-
Change a model characteristic by clicking the pencil icon (
).
.a. Select a new feature list. You cannot change the feature list for the model prepared for deployment because it is a "frozen". (See the FAQ for a workaround to add feature lists.)
b. Change the sample size and optionally enforce a frozen run.
コンプライアンスドキュメント¶
DataRobotは、モデル開発に関連する多くの重要なコンプライアンスタスクを自動化することによって、規制の厳しい業界でデプロイまでの時間を短縮できます。 各モデルに対して個々のドキュメントを生成し、効果的なモデルリスク管理に関する包括的なガイダンスを提供できます。 レポートは、編集可能なMicrosoft Wordドキュメント（.docx）としてダウンロードできます。 生成されたレポートには、規制への準拠要求に応じた適切なレベルの情報および透明性が含まれます。
モデルコンプライアンスレポートは、その形式と内容は規定されていませんが、十分に堅牢なモデル開発、実装、および使用ドキュメントを作成するガイドとして機能します。 ドキュメントは、モデルのコンポーネントが意図したとおりに機能すること、それが意図したビジネス目的に対して適切であること、およびモデルが概念的に堅牢であることを示す証拠を提供します。 このため、レポートは、連邦準備制度理事会のシステムのSR11-7：モデルリスク管理に関するガイダンス完成に役立ちます。Guidance on Model Risk Management（モデルリスク管理に関するガイダンス）への準拠に役立ちます。
To generate a compliance report:
- Select a model from the Leaderboard.
-
From the Model actions dropdown, select Generate compliance report.
-
Workbench prompts for a download location and, once selected, generates the report in the background as you continue experimenting.
Manage experiments¶
At any point after models have been built, you can manage an individual experiment from within its Use Case. Click on the three dots to the right of the experiment name to delete it. To share the experiment, use the Use Case Manage members tool to share the experiment and other associated assets.
