Skip to content

Click in-app to access the full platform documentation for your version of DataRobot.

Time-aware considerations

Both time-aware modeling mechanisms—OTV and automated time series—are implemented using date/time partitioning. Therefore, the date/time partitioning notes apply to all time-aware modeling. See also:

See the documented file requirements for information on file size and series limit considerations.

Note

Considerations are listed beginning with newest additions for easier identification.

Date/time partitioning considerations

  • Frozen thresholds are not supported.

  • Blenders that contain monotonic models do not display the MONO label on the Leaderboard for OTV projects.

  • When previewing predictions over time, the interval only displays for models that haven’t been retrained (for example, it won’t show up for models with the Recommended for Deployment badge).

  • Date/time partitioning is not available for DataRobot Prime.

  • If you configure long backtest durations, DataRobot will still build models, but will not run backtests in cases where there is not enough data. In these case, the backtest score will not be available on the Leaderboard.

  • Timezones on date partition columns are ignored. Datasets with multiple time zones may cause issues. The workaround is to convert to a single time zone outside of DataRobot. Also there is no support for daylight savings time.

  • Dates before 1900 are not supported. If necessary, shift your data forward in time.

  • Leap seconds are not currently supported.

Time series-specific considerations

In addition to the above items, consider the following when working with time series projects:

Accuracy

  • DeepAR:
    • Regression only
    • Feature lists must contain latest naive baseline feature
    • Supports covariates, but only those that are available at prediction time e.g., date derived, known in advance, and calendar features.
    • Target cannot be DND
    • Not available in FW-0 mode
    • Not available in unsupervised mode
    • Training dataset cannot be sampled
  • Temporal hierarchical models:
    • Supports regression projects only
    • Target cannot be DND
    • Not available in FW=0 mode
    • Not available for row-based projects
  • Nowcasting:

    • Because MASE and Theil’s U are only available when the target is derived, these metrics are only available for regression projects with a derived target.
  • Feature Effects, Compliance documentation, and Prediction Explanations are not supported for autoregressive models (Traditional Time Series (TTS) and deep learning models). This includes:

    • All ARIMA:

      • Per Series nonseasonal AUTOARIMA with Fixed Error Terms (required feature flags: Enable Multiseries Scoring Code Developer Blueprints + Enable Scoring Code)
      • Per-Series nonseasonal AUTOARIMA
      • Per-Series nonseasonal AUTOARIMA with Fourier terms
      • Non-seasonal AUTOARIMA
      • AUTOARIMA with naive prediction offset
    • All VAR:

      • Multiseries VARMAX
      • Multiseries VARMAX with Fourier terms
    • All RNN and LSTM (DeepAR, Sequence to Sequence, etc.)

  • Other autoregressive modelers such as Prophet, TBATs, and ETS.

Anomaly Detection

  • Model comparison:

    • External test sets are not available.
    • The “All backtests” option is not available.
  • Multistage OTV is not available for unsupervised projects.

  • The anomaly threshold for the Anomaly Over Time chart is fixed at 0.5 for per-series kind blueprints. Non-per-series blueprints will use a computed threshold, which is dynamic.

  • The Anomaly Assessment Insight:

    • Does not work for unsupervised AutoML
    • The Max number of points is 500 most anomalous per source, but can be reconfigured
    • Is not available for blenders
    • Will not be computed for training if training is considered to be too large.

Data prep tool

Consider the following when doing gap handling and aggregation:

  • Data prep is not supported for deployments or for use with the API.

  • Only numeric targets are supported.

  • Only numeric, categorical, text, and primary date columns are included in the output.

  • The smallest allowed time step for aggregation is one minute.

  • Datasets added to the AI catalog prior to introduction of the data prep tool are not eligible. Re-upload datasets to apply the tool.

  • Shared deployments do not support automatic application of the transformed data prep dataset for predictions.

Data Quality

  • Check for leading-trailing zeros only runs when less than 80% of target values are zeros.

Monotonic constraints

  • XGBoost is the only supported model.

  • While you can create a monotonic feature list after project creation with any numeric post-derivation feature, if you specified a raw feature list as monotonic before project creation, all features in it will be marked as Do not Derive (DND).

  • When there is an offset in the blueprint, for example naive predictions, the final predictions may not be monotonic after offset is applied. The XGBoost itself honors monotonicity.

  • If the model is a collection of models, like per-series XGBoost or performance-clustered blueprint, monotonicity is preserved per series/cluster.

Productionalization

  • Prediction Explanations:

    • Are not available for AutoRegressive Models (LSTM/ARIMA/VARMAX) or blenders containing them.
    • Are defined relative to the training dataset, not the recent history.
    • Require at least 100 rows of validation data.
    • For a model trained into Holdout as part of Autopilot are not available until the holdout is unlocked.
    • For blenders created directly from frozen start/end models trained into Validation are not available. They are available if a blender of the parent models is retrained into Validation or Holdout.
    • Are not supported for series-scaling models in cross-series projects or the blenders containing them.
    • Are only available using the XEMP methodology.
  • ARIMA, LSTM, and DeepAR models cannot be deployed to prediction servers. Instead, deploy using either:

    • the Portable Predictions Server—an execution environment for DataRobot model packages (.mlpkg files).
    • the Make Predictions tab (for datasets up to 1GB).
    • DataRobot Scoring Code (ARIMA only).
  • Scoring code support requires the following feature flags: Enable Scoring Code, Enable Scoring Code Support for Time Series, Enable Scoring Code support for Keras Models (if needed), Enable Multiseries Scoring Code Developer Blueprints (if needed)

  • Time series batch predictions are not available for cross-series projects or traditional time series models (such as ARIMA).

Scale

  • For temporal hierarchical models, the Feature Over Time chart may look different from the data used at the edges of the partitions for the temporal aggregate.

  • When using configurable model parallelization (Customizable FD splits), if one parallel job is deleted during Autopilot, the remaining model split jobs will error.

  • 10GB OTV requires multistep OTV be enabled.

Trust

  • Model Comparison (over time) shows the first 1000 series only. The insight does not support synchronization with job computation status and is only able to show completely precomputed data.

  • Forecast vs Actuals (FvsA) chart:

    • UI is limited to showing and computing a maximum of 100 forecast distances at a time
    • UI is limited to showing 1000 bins at a time
    • API is not public
    • Training CSV export is not available
    • PNG and ZIP export are not available
    • Chart could work slowly on large datasets with wide FDs
    • FvsA chart is not available for projects with [0,0] forecast window
    • Calculation for any particular backtest/source will remove any previously calculated Accuracy Over Time (AOT) data for this backtest/source. However, AOT will be recalculated with FvsA for the selected forecast distance range.
  • Accuracy over Time (AOT) chart:

    • UI is limited to showing 1000 bins at a time
  • When handling data quality issues in Numeric Data Cleansing, some models can experience performance regression.

  • CSV Export is not available for “All Backtest” in the Forecast vs Actuals chart.

Multiseries considerations

In addition to the general time series considerations above, be aware:

  • The Feature Association Matrix is not supported.

  • Most multiseries UI insights and plots support up to 1000 series. For large datasets, however, some insights must be calculated on-demand, per series.

  • Multiseries supports a single (1) series ID column.

  • Multiseries ID values should be either all numeric or all strings. Blank or float data type series ID values are not fully supported.

  • Multiseries does not support Prophet blueprints.

Clustering considerations

  • Clustering is only available for multiseries time series projects. Your data must contain a time index and at least 10 series.

  • At least two clusters must be discovered for the clustering model to be used in a segmented modeling run.

    What does it mean to "discover" clusters?

    To build clusters, DataRobot must be able to group data into two or more distinct groups. For example, if a dataset has 10 series but they are all copies of the same single series, DataRobot would not be able to discover more than one cluster. In a more realistic example, very slight time shifts of the same data will also not be discoverable. If all the data is too mathematically similar that it cannot be separated into different clusters, then it cannot subsequently be used by segmentation.

    The "closeness" of the data is model-dependent—the convergence conditions are different. Velocity clustering would not converge if a project has 10 series, all with the same means. That, however, does not imply that K-means itself wouldn't converge.

    Note, however, the restrictions are less strict if clusters are not being used for segmentation.

Segmented modeling considerations

  • Projects are limited to 100 segments; all segments must total less than 1GB (5GB with feature flag, contact your DataRobot representative).

  • Predictions are only available when using the Make Predictions tab on the Combined Model's Leaderboard or via the API.

    • If you manually assigned segments by selecting a segment ID (instead of using discovered clusters), the prediction dataset must not contain a new segment ID that does not appear in the training dataset.
    • The prediction dataset must fulfill historical data requirements for each segment. For segment projects with detected seasonality, there must be more historical rows than those segment projects without seasonality.
  • Time series clustering projects are supported. See the associated considerations.

Release 6.0 and earlier

  • For the Make Predictions tab:

    • The Forecast Settings modal appears only if the dataset was uploaded after release 5.3. The automatically generated extended prediction file template is available only if the dataset was uploaded after release 6.0.

    • If a dataset exceeds the upload file size limit after expansion it will not be expanded.

    • When a prediction dataset requires automatic expansion and also contains rows without a target, the expanded rows might have duplicate dates in the rows without target (and will fail to predict). To resolve this, simply remove the rows without target before uploading the file.

    • DataRobot displays a warning when KA values are missing but does not itemize the specific missing values per forecast point.

  • Classification models are not optimized for rare events, and should have >15% frequency for their minority label.

  • Run Autoregressive models using the "Baseline Only" feature list. Using other feature lists could cause Feature Effects or compliance documentation to fail, as the autoregressive models do not use the additional features that are part of the larger default lists and they are not designed to work with them.

  • Feature Effects and Compliance documentation are disabled for LSTM/DeepAR blueprints.

  • Eureqa with Forecast Distance is limited to 15 FD values. They will only run on smaller datasets with fewer than 100K rows or if the total number of levels for the categorical features is less than 1000. Their grid search plots in Advance Tuning marks only the single best grid search point, independent of the FD value. The blueprint can take a long time to complete if the task size parameter is set too large.

  • Forecast distance blenders are limited to projects with a maximum of 50 FDs.

  • The "Forecast distance" selector on the Coefficients tab is not available for backtests and models that do not use ForecastDistanceMixin, for example, ARIMA models.

  • Monthly differencing on daily datasets can only be triggered through detection. Currently, there is no support to specify monthly seasonality via an advanced option in the UI or API.

  • RNN-based (LSTM and GRU—long short-term memory and gated recurrent unit) supports a maximum categorical limit of 1000 (to prevent OOM errors). High-cardinality features will be truncated beyond this.

  • The training partition for the holdout row in the flexible backtesting configuration is not directly editable. The duration of the first backtest’s training partition is used as the duration for the training partition of the holdout.

  • For Repository blueprints, selecting a best-case default feature list is available for ARIMA models only.

  • Hierarchical modeling requires the data’s series to be aligned in time (specifically 95% of series must appear on 95% of the timestamps in the data).

  • Hierarchical and series-scaled blueprints require the target to be non-negative.

  • Series-scaled blueprints only support squared loss (no log link).

  • Hierarchical and LSTM blueprints do not support projects that require sampling.

  • Model-per-series blueprints (XGB, XGB Boost, ENET) support up to 50 series. They will not be advance tunable if number of series is more than 10.

  • ARIMA per-series blueprints are limited to 15K rows per series (i.e., 150K rows for 10 series) and support up to 40 series. The blueprint runs in Autopilot when the number of series is less than 10. Due to a refit for every prediction, the series accuracy computation can take a long time.

  • Clustered blueprints are not available for classification. Similarity-based clustering is very time-consuming and can take a long time to train and will use large amounts of memory (use the default performance-based clustering for large datasets).

  • Zero-inflated blueprints are enabled if the target’s minimum value is 0.

  • Zero-inflated blueprints only support the “nonzero average baseline” feature list.

  • Setting the target to do-not-derive still derives the simple naive target feature for regression projects.

  • Hierarchical and zero-inflated models cannot be used when a target is set to do-not-derive because the feature derivation process does not generate the target derived features required for zero-inflated & hierarchical models.

  • The group ID for cross-series features cannot have blank or missing values; they cannot mix numeric and non-numeric values, similar to the series ID constraints.

  • Prediction Explanations are not available for XGBoost-based hierarchical and two-stage models.

  • Series scaling blueprints may have poor accuracy when predicting new series.

  • The Feature Association Matrix is not supported in multiseries projects.

  • Timestamps can be irregularly spaced but cannot contain duplicate dates within a series.

  • Time series datasets cannot contain dates past the year 2262.

  • To ensure backtests have enough rows, in highly irregular datasets use the row-count instead of duration partitioning mode.

  • VARMAX and VAR blueprints do not support log-transform/exponential modeling.

  • ARIMA, VARMAX, and VAR blueprint predictions require history back to the end of the training data when making predictions.

  • For non-forecasting time series models (those that allows predicting the current target FW=[0, 0]):

    • Forecast window FW=[0,0] is allowed but not FW=[0, N] where N>0
    • Forecast window FW=[0,0] will not generate any lags of the target (similar to OTV)
  • Loss families have changed for time series blenders, which may slightly change blending results. Specifically:

    • When the target is exponential and metric is RMSE, MASE, or Theil's U, the loss family is Poisson or Gamma.
    • When the target is not exponential, the loss family is Gaussian.
  • Binary classification projects have somewhat different options available than regression projects. Additionally, classification projects:

    • are not optimized for rare events (they should have >15% frequency of the minority label).
    • must have examples of all labels in all backtest partitions.
    • do not support differencing, ARIMA, or detecting seasonality.
    • can show error bars beyond the 0-1 range in the prediction preview plot.
  • Millisecond datasets:

    • Can only specify training and partitioning boundaries at the second level.
    • Must span multiple seconds for partitioning to work.
  • Row-based projects require a primary date column.

  • Calendar event files:

    • cannot be updated in an active project. You must specify all future calendar events at project start or if you did not, train a new project.
    • If you upload a multiseries calendar, changing the series ID after the upload will require you to clear and re-upload the dataset.
    • must be under 10MB.
  • When running blueprints from the repository, the Time Series Informative Features list (the default selection if you do not override it) is not optimal. Preferably, select one of the “with differencing” or the “no differencing” feature lists.

  • The Forecast Window must be 1000 forecast distances (FDs)/time steps or fewer for small datasets.

  • Automated Time Series does not support DataRobot Prime

  • You cannot modify R code for Prophet blueprints; also, they do not support calendar events and cannot use known in advance features.

  • Only Accuracy Over Time, Stability, Forecasting Accuracy, and Series Insights plots are available for export; other time series plots are not exportable from the UI or available through the public API.

  • Large datasets with many forecast distances are down-sampled after feature derivation to <25GB.

  • Accuracy Over Time training computation is disabled if the dataset exceeds the configured threshold after creation of the modeling dataset. The default threshold is 5 million rows.

  • Seasonal AUTOARIMA uses large amounts of memory for large seasonality and, due to Python 2.7 issues, could fail on large datasets.

  • Seasonality is only detected automatically if the periodicity fits inside the feature derivation window.

  • TensorFlow neural network blueprints (in the Repository) do not support text features or making predictions on new series not in the training data.


Updated January 31, 2023
Back to top