Experiment insights¶

Tile	Description
	Opens insights that provide experiment-level information for all models.

Experiment insights are tools that provide contextual information for the model:

• Use Learning Curves to compare model performance across different sample sizes.
• Use Speed vs Accuracy to graph tradeoffs between runtime and predictive accuracy.

Learning Curves¶

Learning Curves help to determine whether it is worthwhile to increase the size of the dataset. Getting additional data can be expensive, but may be worthwhile if it increases model accuracy. The graph illustrates, for top-performing models, how model performance varies as the sample size changes. A metric dropdown is available to sort the results, independently of the sort setting for the Leaderboard. It charts how well a model group performs when it is computed across multiple sample sizes in the training and validation partitions. This grouping represents a line in the graph, with each dot on the line representing the sample size and score of an individual model in that group.

View values¶

To see the values for a curve point, you can mouse over or click. The corresponding model highlights in the model list.

Single point curveLineModel listing on hoverModel(s) selected

Filter view¶

You can filter the Learning Curve display by metric or feature list. Select a metric to compare results. For example, the image below shows results for LogLoss and FVE Binomial:

By default, the graph plots using the Informative Features feature list. Filter the graph to show models for a different feature list, including custom lists (if it was used to run models) by using the Feature list dropdown menu. The menu lists all feature lists that belong to the project. If you have not run models on a feature list, the option is displayed but disabled.

Compute new sample sizes¶

Because Quick Autopilot uses one-stage training, the Learning Curves graph will initially populate with only a single point for each of the top 10 performing models. Use the Compute Learning Curves option to increase the display points.

To compute new points:

1. Select which models to compute for from the list on the right side of the window. Selected models highlight in the display. Click Compute Learning Curves.

2. Add or remove sample sizes, including custom sizes.

   

3. Click Compute.

Interpret Learning Curves¶

Consider the following when evaluating the Learning Curves graph:

• Study the model for any sharp changes or performance decrease with increased sample size. If the dataset or the validation set is small, there may be significant variation due to the exact characteristics of the datasets.

• Model performance can decrease with increasing sample size, as models may become overly sensitive to particular characteristics of the training set.

• In general, high-bias models (such as linear models) may do better at small sample sizes, while more flexible, high-variance models often perform better at large sample sizes.

• Preprocessing variations can increase model flexibility.

Speed vs Accuracy¶

Predictive accuracy often comes at the price of increased prediction runtime. The Speed vs Accuracy analysis plot shows the tradeoff between runtime and predictive accuracy to help choose the best model with the lowest overhead. The display is based on the validation score, using the currently selected metric.

• The Y-axis lists the metric currently selected on the Leaderboard. Use the Metric dropdown to change metric.

• The X-axis displays the estimated time, in milliseconds, to make 1000 predictions. Total prediction times include a variety of factors and vary based on the implementation. Mouse over any point on the graph, or the model name in the legend to the right, to display the estimated time and the score.