Time series data wrangling¶

Feature engineering is a major component of successfully addressing use case requirements. By wrangling—creating recipes of operations and applying them first to a sample and then, when verified, to a full dataset—time-aware data, you can perform time series feature engineering during the data preparation phase. Executing operations like lags and rolling statistics on input data provides control over which time-based features are generated before modeling. By reviewing the preview that results from adding both time-aware and non-time-aware operations, you can adjust before publishing, preventing the need to rerun modeling if what would otherwise be done automatically doesn't fit your use case.

This page describes the workflow specific to time series wrangling. See also the full data wrangling documentation for more information.

基本的なワークフロー¶

The following describes the basic workflow:

1. Connect to Snowflake, using a configured data connection.
2. From Preview settings, configure a live sample using date/time as the sampling method.
3. When sampling settings are complete, click Resample to apply the settings.
4. Add operations from the Recipes panel.

Set date/time sampling¶

By default, DataRobot retrieves 10000 random rows for the live sample. To use time series wrangling, and more specifically, to configure DataRobot to create a derivation plan for your sample, you must use the date/time sampling method. That is, while you can apply lags and rolling statistics to non-date/time samples, you cannot take advantage of the automation provided by the suggested derivation plan.

To configure date/time sampling, open Preview settings and set the Sampling method to date/time.

Then, complete the date/time-specific fields described in the table below:

すべてのフィールドに入力してリサンプリングをクリックします。 The live sample updates to display the data selected based on the configuration. You can now add operations.

操作の追加¶

Operations are the transformations that will be applied to the source data to prepare it for modeling. The sections below describe the time-aware operations; see the operation reference for information on other operations and working with operations within a recipe. Click + Add operations to begin configuring transformations.

The Lag features and Derive rolling statistics operations can result in more than one feature added to the dataset. The Derive time series features operation, on the other hand, substitutes the input dataset with an output dataset that is expanded by the specified forecast distances for each operation.

When all fields are complete for a given operation, click +Add to recipe. The preview updates based on your changes. A summary of each operation is provided in the Recipe panel.

Derive time series features¶

When working with time series wrangling, the data is ordered by the specified date column (the "ordering feature") and partitioned by series ID column, if applicable. The feature derivation windows (FDW) are defined in terms of rows. In those windows, the window start is excluded and the end is included. So, the dataset is "reordered" according to date and then, for example, if FDW=5, five rows are used to derive features.

The Derive time series features operation generates new time series features. To start, you supply fundamental information and DataRobot then expands the data according to forecast distances, adds known in advance columns (if specified) and naive baseline features, and then replaces the original sample. To create new features, you can have DataRobot suggest a derivation plan and automatically create new features, manually add tasks for up to 200 features, or apply a hybrid approach.

Configure parameters¶

To configure parameters that set up the ability to use the automated derivation plan, after setting the sampling method to date/time, select the Derive time series features operation. Alternatively, if the sampling method is random or N-rows, you can configure time series wrangling parameters but cannot take advantage of the automation.

The following table describes the parameters to configure:

Note that all setting are summarized in the right panel summary. When all fields are complete and correct, click Next.

派生のための特徴量¶

After clicking Next, the Features for derivation panel opens.

The following options are available for creating derived time series features:

Suggest derivation plan¶

Use the Suggest derivation plan option when you have set the sampling method to date/time and want DataRobot to create lags and rolling statistics based on the configuration settings you supply. Click the option and complete the fields:

* The forecast point is part of a feature derivation window; the feature derivation window end is always zero. For this reason, there is no blind history gap in time series data wrangling.

Once all fields are completed, select Create plan. DataRobot applies operations (tasks) for features.

Click on a feature name in the left panel to see the tasks created reflected in the middle panel. From there, click Add derivation task to add more tasks to the feature. Click Delete task to remove any tasks you do not want eventually applied to the modeling dataset. In the right panel, tasks-per-feature are added to the summary.

Review the plan:

• If you are satisfied with the derivation plan, click Add to recipe. ライブサンプルは、DataRobotがデータソースから新しいサンプルを取得し、操作を適用すると更新され、変換をリアルタイムで確認できます。
• If you are not satisfied, click Suggest derivation plan and reset the configuration to different values. The original output will be overwritten by the output from the new plan.

Add features manually¶

To add features manually, regardless of the sampling method, configure parameters and the toggle on the option to Add features manually. Click in the entry box to select a feature to derive and then click Add.

The feature is added to the list in the left panel and derivation configuration becomes available in the center panel. Or, click on any previously added feature in the left panel to open the derivation configuration, for example, if you want to revisit the configuration, and add or remove tasks.

To add a task, from the Task dropdown, make a selection. Each task has its own configuration fields:

Click Add derivation task to add more tasks or delete task to remove an individual task from the derivation configuration. As you work with features, DataRobot reports the number of tasks (including incompletely configured tasks) in the right and left panels.

When all derivation tasks are configured for a feature, click Add to recipe. ライブサンプルは、DataRobotがデータソースから新しいサンプルを取得し、操作を適用すると更新され、変換をリアルタイムで確認できます。

Hybrid approach¶

With sampling set to date/time, you can use a hybrid approach that leverages the automation of Suggest derivation plan while allowing you to add features that may have been missed due to your configuration. You can also add tasks to features that were transformed.

To add new features after the automated plan creates transformations, either:

• Toggle on Add features manually to search for features and then configure tasks.
• Click a feature in the left panel to show the task configuration and click Add derivation task to set new tasks for that feature.

When all feature transformation instructions are complete, click Add to recipe. ライブサンプルは、DataRobotがデータソースから新しいサンプルを取得し、操作を適用すると更新され、変換をリアルタイムで確認できます。

Lag features¶

A lag represents a specific time period between an occurrence and its impact and are important for capturing delayed relationships between features in time series data. The lag measurement (e.g., 3, 7) is implemented based on the time step that is detected in the data. A lag is calculated relative to the current row—the first lag is the previous row, the second lag is two rows back, and so on. Click Lag features to set the lag configuration.

次の表にフィールドを示します。

Derive rolling statistics¶

Rolling statistics allows you to calculate statistics for a rolling window that is comprised of a specified number of rows. They can be created for both numeric and categorical features.

Output of the operation is:

• Numeric: One or several columns, depending on the statistical method specified.
• Categorical: One column, calculated in a window including the current row

The following table describes the fields for both numeric and categorical rolling statistics:

Specify a UDF¶

Both options provide an optional input field where you can provide a path to user-defined functions (UDFs), which are available as SQL scripts on GitHub. They include user-defined functions/aggregations that compute rolling median and most frequent statistics for different databases. DataRobot recommends using these functions when wrangling with time series operations. They generate SQL that is smaller and faster, without needing additional joins to create windows.

To use the UDFs, download the scripts locally and then provide them in the Rolling median.. or Rolling most frequent user-defined function field, in the format catalog.schema.udf_name.

注意事項¶

• For time series wrangling, only Snowflake connections are supported in the UI. Postgres connections are available via the API. Other data sources are under development.

• The maximum number of output features is 200.

• Only regression experiments (numeric targets) are supported.

• Only one derivation plan is allowed. You can add features manually to the plan output if more are needed.

• Windows defined in terms of time units—for example, days and minutes—are not supported.

• There is no validation of the data quality on the input dataset.

次のステップ¶

ここから、次のことができます。

• レシピをデータソースにパブリッシュし、新しい出力データセットを生成します。