Run a pipeline¶
As you add a module to a pipeline and configure it, DataRobot compiles the module. If the module compiles successfully, you can run it. Otherwise, a compilation error displays in the Console tab.
Compilation errors can be simple errors like missing sufficient ports or unconnected ports. In this case, it's an unconnected port error.
To learn about errors and warnings generated during compilations and runs, see Module status.
You can run a module explicitly using the Run to selection command. You can also run a module as part of a pipeline path, for example, by applying Run to selection on a downstream module.
Select a tab below to learn how to run pipelines.
In the workspace editor, select the module the pipeline will run to and click the Run to selection button above the pipeline.
View the output of the module run in the Results tab.
Following are the results of a successful run.
If a run is unsuccessful, an error displays in the Console tab.
If you run a pipeline to a selected module and an error occurs, with the module still selected, navigate to the Console tab under the workspace to view its details. Runtime errors can be caused by issues like invalid SQL statements and invalid file paths. See Module status for a description of the status types.
To run an entire pipeline, close the workspace editor and click Run on the top right of the workspace.
The Run button displays whether you're viewing the Info, Pipelines, Run Schedule and History, or Comments tab.
To see the results, select the Run Schedule and History tab in the workspace.
You can also schedule pipelines to run at regular intervals.
Download a preview¶
When you run to a selected module, you see a preview of the results at that point in the pipeline. The preview displays in the Results tab:
Click Download Preview to download the results preview.
A status icon displays beside each module to indicate the current state of the module, for example, compiled, running, and successful run.
In this example, the CSV Reader module compiles but the Spark SQL module fails to compile. The Console tab displays the errors.
The following table describes each status.
|Module has been added to the pipeline, but has not compiled yet.|
|Module has been compiled successfully.|
|Module compilation failed. Check the Console tab for error details.|
|Module is running or waiting for a dependency to finish running. The status icon spins to indicate the module is running.|
|Module is queued and will run once resources become available.|
|Module has run successfully. Check the Results tab for details. If you run this module again, it will intelligently use the cached results and will not re-run. To force a re-run of the module, use Force Run from the module actions menu.|
|Module run failed. Select the failed module and check the Console tab for error details. You can make changes to the module configuration or edit the specification and rerun the module.|
Schedule a pipeline run¶
A powerful feature of DataRobot Pipelines is the ability to configure your pipeline and run it at regular intervals. This allows you to automate data updates. To schedule a pipeline run:
Navigate to a workspace, select the Run Schedule and History tab, and click Set schedule.
Set the frequency and time you want the workspace pipeline to run.
To set more granular times, click Use advanced scheduler. The advanced scheduler lets you schedule runs for specific minutes, hours, days of the month, months, and days of the week.
Select the Activate schedule after it is saved checkbox to turn the schedule on now. You can uncheck it if you want to wait and enable the schedule later.
The schedule is set.
To deactivate the schedule, click pause (). You can later enable the schedule.
Enable a paused schedule¶
If you created a schedule but chose not to activate it right away or if you deactivated a schedule, it appears as "Paused."
To later enable the schedule, click play ().
Edit a saved schedule¶
To edit a saved schedule, do one of the following:
- Click pause () to deactivate the schedule temporarily. You can later enable the schedule.
- Click the pencil icon () to update your schedule settings.
- Click delete () to delete the schedule.
View run history¶
Once your pipeline runs, you can view the run history:
Click a run to view a log file and the pipeline.yml code.