Dataset requirements¶

This section provides information on dataset requirements:

• General requirements
• Ensuring acceptable file sizes
• AutoML file import sizes
• Time series (AutoTS) file import sizes
• Feature Discovery file import sizes
• Pipeline data requirements
• File formats
• Encodings and character sets
• Special column detection
• Length and name conversion
• File download sizes

See the associated considerations for important additional information.

General requirements¶

Consider the following dataset requirements for AutoML, time series, and Visual AI projects. See additional information about preparing your dataset for Visual AI.

Ensure acceptable file import size¶

When ingesting a dataset, its actual on-disk size might be different inside of DataRobot.

• If the original dataset source is a CSV, then the size may differ slightly from the original size due to data preprocessing performed by DataRobot.

• If the original dataset source is not a CSV (e.g., is SAS7BDAT, JDBC, XLSX, GEOJSON, Shapefile, etc.), the on-disk size will be that of the dataset when converted to a CSV. SAS7BDAT, for example, is a binary format that supports different encoding types. As a result, it is difficult to estimate the size of data when converted to CSV based only on the input size as a SAS7BDAT file.

• XLSX, due to its structure, is read in as a single, whole document which can cause OOM issues when trying to parse. CSV, by contrast is read in in chunks to reduce memory usage and prevent errors. Best practice recommends not exceeding 150MB for XLSX files.

• If the original dataset source is an archive or a compressed CSV (e.g., .gzip, .bzip2, .zip, .tar, .tgz), the actual on-disk size will be that of the uncompressed CSV after preprocessing is performed.

Keep the following in mind when considering file size:

• Some of the preprocessing steps that are applied consist of converting the dataset encoding to UTF-8, adding quotation marks for the field data, normalizing missing value representation, converting geospatial fields, and sanitizing column names.

• In the case of image archives or other similar formats, additional preprocessing will be done to add the images file contents to the resulting CSV. This potentially will make the size of the final CSV drastically different from the original uploaded file.

• File size limitations are applied to files once they have been converted to CSV. If you upload a zipped file into DataRobot, when DataRobot extracts the file, the file must be less than the file size limits.

• If a delimited-CSV dataset (CSV, TSV, etc.) size is close to the upload limit prior to ingest, it is best to do the conversion outside of DataRobot. This helps ensure that the file import does not exceed the limit. If a non-comma delimited file is near to the limit size, it may be best to convert to a comma-delimited CSV outside of DataRobot as well.

• When converting to CSV outside of DataRobot, be sure to use commas as the delimiter, newline as the record separator, and UTF-8 as the encoding type to avoid discrepancies in uploaded file size and size counted against DataRobot's maximum file size limit.

• Consider modifying optional feature flags in some cases:

  • Disable Early Limit Checking: By selecting, you disable the estimate-based early limit checker and instead use an exact limit checker. This may help allow ingestion of files that are close to the limit in case the estimate is slightly off. Note however that if the limit is exceeded, projects will fail later in the ingest process.
  • Enable Minimal CSV Quoting: Sets the conversion process to be more conservative when quoting the converted CSV, allowing the CSV to be smaller. Be aware, however, that doing so may make projects non-repeatable. This is because if you ingest the dataset with and without this setting enabled, the EDA samples and/or partitioning may differ, which can lead to subtle differences in the project. (By contrast, ingesting the same dataset with the same setting will result in a repeatable project.)

AutoML file import sizes¶

The following sections describe supported file import sizes.

Beyond 10GB ingest (SaaS only)¶

Ingest of up to 20GB training datasets provides large-scale modeling capabilities. When enabled, file ingest limit is increased from 10GB to 20GB.

Consider the following when training with 20GB:

• Available for binary classification and regression projects only.
• No support for Visual AI or Location AI projects.
• Ingestion is only available from an external source (data connection or URL); training data must be registered in the AI Catalog (20GB datasets cannot be directly uploaded from a local computer).
• Sliced insights are disabled.
• Feature Discovery is disabled.
• By default, Feature Effects generates insights for the top 500 features (ranked by feature impact). With projects greater than 10GB, in consideration of runtime performance, Feature Effects generates insights for the top 100 features.

OTV requirements¶

For out-of-time validation (OTV) modeling, maximum dataset size: Less than 5GB.

OTV backtests require at least 20 rows in each of the validation and holdout folds and at least 100 rows in each training fold. If you set a number of backtests that results in any of the runs not meeting that criteria, DataRobot only runs the number of backtests that do meet the minimums (and marks the display with an asterisk). For example:

• With one backtest, no holdout, minimum 100 training rows and 20 validation rows (120 total).
• With one backtest and holdout, minimum 100 training rows, 20 validation rows, 20 holdout rows (140 total).

Prediction file import sizes¶

Time series file requirements¶

When using time series, datasets must be CSV format and meet the following size requirements:

If you set a number of backtests that results in any of the runs not meeting that criteria, DataRobot only runs the number of backtests that do meet the minimums (and marks the display with an asterisk). Specific features of time series:

* Self-Managed AI Platform versions 5.0 or later are limited to 100,000 series; versions 5.3 or later are limited to 1,000,000 series.

For releases 4.5, 4.4 and 4.3, datasets must be less than 500MB. For releases 4.2 and 4.0, datasets must be less than 10MB for time series and less than 500MB for OTV. Datasets must be less than 5MB for projects using Date/Time partitioning in earlier releases.

Feature Discovery file import sizes¶

When using Feature Discovery, the following requirements apply:

• Secondary datasets must be either uploaded files or JDBC sources registered in the AI Catalog.

• You can have a maximum of 30 datasets per project.

• The sum of all dataset sizes (both primary and secondary) cannot exceed 100GB, and individual dataset sizes cannot exceed 11GB. See the download limits download limits mentioned below.

Data formats¶

DataRobot supports the following formats and types for data ingestion. See also the supported data types.

File formats¶

• .csv, .dsv, or .tsv* (preferred formats)
• database tables
• .xls/.xlsx
• PDF**
• .sas7bdat
• .parquet
• .avro**

*The file must be a comma-, tab-, semicolon-, or pipe-delimited file with a header for each data column. Each row must have the same number of fields, some of which may be blank.

**These file types are preview. Contact your DataRobot representative for more information.

Location AI file formats¶

The following Location AI file types are supported only if enabled for users in your organization:

• ESRI Shapefiles
• GeoJSON
• ESRI File Geodatabase
• Well Known Text (embedded in table column)
• PostGIS Databases (The file must be a comma-delimited, tab-delimited, semicolon-delimited, or pipe-delimited file and must have a header for each data column. Each row must have the same number of fields (columns), some of which may be blank.)

Compression formats¶

• .gz
• .bz2

Archive format¶

• .tar

Compression and archive formats¶

• .zip
• .tar.gz/.tgz
• .tar.bz2

Both compression and archive are accepted. Archive is preferred, however, because it allows DataRobot to know the uncompressed data size and therefore to be more efficient during data intake.

Decimal separator¶

The period (.) character is the only supported decimal separator—DataRobot does not support locale-specific decimal separators such as the comma (,). In other words, a value of 1.000 is equal to one (1), and cannot be used to represent one thousand (1000). If a different character is used as the separator, the value is treated as categorical.

A numeric feature can be positive, negative, or zero, and must meet one of the following criteria:

• Contains no periods or commas.
• Contains a single period (values with more than one period are treated as categorical).

The table below provides sample values and their corresponding variable type:

Encodings and character sets¶

Datasets must adhere to the following encoding requirements:

• The data file cannot have any extraneous characters or escape sequences (from URLs).

• Encoding must be consistent through the entire dataset. For example, if a datafile is encoded as UTF-8 for the first 100MB, but later in the file there are non-utf-8 characters, it can potentially fail due to incorrect detection from the first 100MB.

Data must adhere to one of the following encodings:

• ascii
• cp1252
• utf-8
• utf-8-sig
• utf-16
• utf-16-le
• utf-16-be
• utf-32
• utf-32-le
• utf-32-be
• Shift-JIS
• ISO-2022-JP
• EUC-JP
• CP932
• ISO-8859-1
• ISO-8859-2
• ISO-8859-5
• ISO-8859-6
• ISO-8859-7
• ISO-8859-8
• ISO-8859-9
• windows-1251
• windows-1256
• KOI8-R
• GB18030
• Big5
• ISO-2022-KR
• IBM424
• windows-1252

Special column detection¶

Note that these special columns will be detected if they meet the criteria described below, but currency, length, percent, and date cannot be selected as the target for a project. However, date can be selected as a partition feature.

Date and time formats¶

Columns are detected as date fields if they match any of the formats containing a date listed below. If they are strictly time formats, (for example, %H:%M:%S) they are detected as time. See the Python definition table for descriptions of the directives. The following table provides examples using the date and time January 25, 1999 at 1:01 p.m. (specifically, 59 seconds and 000001 microseconds past 1:01 p.m.).

Percentages¶

Columns that have numeric values ending with % are treated as percentages.

Currencies¶

Columns that contain values with the following currency symbols are treated as currency.

• $
• EUR, USD, GBP
• £
• ￡ (fullwidth)
• €
• ¥
• ￥ (fullwidth)

Also, note the following regarding currency interpretation:

• The currency symbol can be preceding ($1) or following (1EUR) the text but must be consistent across the feature.
• Both comma (,) and period (.) can be used as a separator for thousands or cents, but must be consistent across the feature (e.g., 1000 dollars and 1 cent can be represented as 1,000.01 or 1.000,01).
• Leading + and - symbols are allowed.

Length¶

Columns that contain values matching the convention <feet>’ <inches>” are displayed as variable type length on the Data page. DataRobot converts the length to a number in inches and then treats the value as a numeric in blueprints. If your dataset has other length values (for example, 12cm), the feature is treated as categorical. If a feature has mixed values that show the measurement (5m, 72in, and 12cm, for example), it is best to clean and normalize the dataset before uploading.

Column name conversions¶

During data ingestion, DataRobot converts the following characters to underscores (_): -, $, . {, }, ", \n, and \r.

File download sizes¶

Consider the following when downloading datasets:

• There is a 10GB file size limit.
• Datasets are downloaded as CSV files.
• The downloaded dataset may differ from the one initially imported because DataRobot applies the conversions mentioned above.