Worker Queue¶
The Worker Queue, displayed in the right-side panel of the application, is a place to monitor the steps of EDA1 and EDA2 and set the number of workers used for model building:
After modeling is complete, you can rerun Autopilot at the next level ("Get More Accuracy"), configure modeling settings and rerun the project, and unlock project holdout.
Minimize and expand the panel
Move your mouse over the edge between the Worker Queue and the Leaderboard to expose an arrow which you can click to expand or minimize the queue:
See the section on troubleshooting workers for help when you cannot add workers.
Understand workers¶
DataRobot uses different types of workers for different types of jobs, such as:
- Uploading data
- Computing EDA
- Training a model
- Creating insights
The workers responsible for managing data and computing EDA are shared across an organization. They are a pool of resources to ensure that all shared services are highly available. Modeling workers, on the other hand, are allocated to each user by their administrator. The following sections describe the Worker Queue for modeling and prediction workers.
Monitor build progress¶
After you start building, all running and pending models appear in the Worker Queue.
The queue prioritizes tasks based on the mechanism of the task’s submission. It starts processing “more important” jobs first, assigning all tasks submitted by Autopilot to a lower (background) priority. All other tasks (those not started by Autopilot) are assigned a default priority. For example, user-submitted tasks such as computing Feature Impact are processed by the next available worker without having to wait for Autopilot to finish building models.
You can do the following in the Worker Queue:
Display | Description |
---|---|
View progress | See summary and detailed information about each model as it builds. |
Adjust workers | Adjust the number of simultaneous workers used for the current build. |
Pause processing | Pause the queue while allowing processing models to complete. |
Stop processing | Stop the queue, removing all processing and queued models from the project. |
Reorder builds | Reorder queued models. |
Cancel builds | Remove scheduled builds from the queue. |
Restart model runs | Select options to rerun models with different build criteria. |
If a worker or build fails for any reason, DataRobot lists and records the event under Error at the bottom of the queue. Additionally, the event is reported in a dialog and recorded and available from the Manage Projects inventory.
View progress¶
The Worker Queue is broken into two parts—models in progress and models in queue. For each in-progress model, DataRobot displays a live report of CPU and RAM use.
You can expand and collapse the list of queued models by clicking the arrow next to the queue name:
Adjust number of workers¶
You can adjust the number of workers that DataRobot uses for model building, up to the maximum number allowed for you (set by your administrator). Note that if a project was shared with you and the owner has a higher allotment, the allowed worker total displays the owners allotment, which is not necessarily what you are allowed. To increase or decrease the number of workers, click the orange arrows:
Pause the Worker Queue¶
To pause the Worker Queue, click the pause symbol (double vertical bars) at the top of the queue. After pausing, the symbol changes to a play symbol (arrow).
When you pause the queue, processing continues on in-progress models. As those models complete, workers become available for the next queued model. The position is not filled until you un-pause the queue. Click the play arrow to resume building models.
Stop the Worker Queue¶
To stop the Worker Queue, click the "X" symbol at the top of the queue.
When you stop the queue, all in-progress or queued models are immediately removed, and the modeling process ends.
Reorder workers¶
To prioritize training specific models, you can drag-and-drop queued models to a new position.
If a job triggers dependencies, you cannot reorder the queue so that a model's dependencies are trained after the initial model. Attempting to do so returns the model to its original position in the queue. For example, launching job A
creates two dependencies, jobs B
and C
. Because jobs B
and C
are required to build job A
, you cannot move job A
ahead of them in the Worker Queue.
Note that you cannot reorder in-process models.
Cancel workers¶
You can remove an in-process or queued model by clicking the X next to the model name. An in-process model is immediately cancelled, a queued model is removed from the queue.
Restart a model build¶
When your build completes, or if it is paused, you can use the same data to start a new build. Restart your build from either:
- The Worker Queue, using the Configure modeling settings link.
-
The Data page's Feature Lists tab.
Clicking the link opens a dialog box where you can select which feature list to use. Note these considerations when selecting a list and retraining. Then, click Restart Autopilot to begin the new build.
Troubleshoot workers¶
If you expect to be able to increase your worker count but cannot, the reasons may be:
- You have hit your worker limit.
- Your workers are part of a shared pool.
- Your workers are in use by another project.
Worker limit¶
Modeling worker allocations are set by your administrator. Each worker processes a modeling job. This job count applies across all projects in the cluster, that is, multiple browser windows building models are all a part of your personal worker count—more windows does not provide more workers.
Pooled workers¶
If you are in an organization, it may implement a shared pool of workers. In this case, workers are allocated across all jobs for all users in the organization on a first-in, first-out basis. While you may not have to wait for jobs of other users in your organization to complete, your (and their) jobs will be seeded in the queue and processed as they were received.
Workers in use¶
If you believe you should be able to increase the worker count but you cannot, for example, "using X of Y workers," there are two values to consider for debugging. When Y is lower than you expect, check your worker limit and the org limit. When X is less than you expect, check whether workers are being allocated to other projects or users in your organization.
To check worker use in your projects, navigate to the Manage Projects inventory and look for queued jobs. You can identify them by:
-
An icon and count in the inventory.
-
The list that results from using Filter Projects and selecting Running or queued.
If you find a project with queued jobs, you can stop Worker Queue processing.
- Click on the project in the inventory to make it the active project.
-
In the Worker Queue, click the X icon () to remove all tasks. This removes all in-progress or queued models.
You can also pause the project. As workers complete active jobs, they will become available to pick up jobs from other projects.
Note
When you delete a project using the Actions menu in the inventory, all its worker jobs are automatically terminated.