Skip to content
For Managed AI Platform users, see the cloud platform documentation
Developer documentation > API reference > Python API client > Data preparation > Data registry

Datasets

Dataset

class datarobot.models.Dataset

Represents a Dataset returned from the api/v2/datasets/ endpoints.

Variables

Attribute Type Description
id string The ID of this dataset
name string The name of this dataset in the catalog
is_latest_version bool Whether this dataset version is the latest version of this dataset
version_id string The object ID of the catalog_version the dataset belongs to
categories list(string) An array of strings describing the intended use of the dataset. The supported options are “TRAINING” and “PREDICTION”.
created_at string The date when the dataset was created
created_by string, optional Username of the user who created the dataset
is_snapshot bool Whether the dataset version is an immutable snapshot of data which has previously been retrieved and saved to Data_robot
data_persisted Optional[bool] If true, user is allowed to view extended data profile (which includes data statistics like min/max/median/mean, histogram, etc.) and download data. If false, download is not allowed and only the data schema (feature names and types) will be available.
is_data_engine_eligible bool Whether this dataset can be a data source of a data engine query.
processing_state string Current ingestion process state of the dataset
row_count Optional[int] The number of rows in the dataset.
size Optional[int] The size of the dataset as a CSV in bytes.
sample_size dict, optional The size of data fetched during dataset registration. For example, to fetch the first 95 rows, the sample_size value is {‘type’: ‘rows’, ‘value’: 95}. Currently only ‘rows’ type is supported.

get_uri()

method get_uri()

Returns

Returns Description
url Permanent static hyperlink to this dataset in AI Catalog.

Return type: str

upload()

classmethod upload()

This method covers Dataset creation from local materials (file & DataFrame) and a URL.

Parameters

Parameter Type Description
source str, pd.DataFrame or file object Pass a URL, filepath, file or DataFrame to create and return a Dataset.

Returns

Returns Description
response The Dataset created from the uploaded data source.

Return type: Dataset

Raises

Exception Description
InvalidUsageError If the source parameter cannot be determined to be a URL, filepath, file or DataFrame.

Examples

# Upload a local file
dataset_one = Dataset.upload("./data/examples.csv")

# Create a dataset via URL
dataset_two = Dataset.upload(
    "https://raw.githubusercontent.com/curran/data/gh-pages/dbpedia/cities/data.csv"
)

# Create dataset with a pandas Dataframe
dataset_three = Dataset.upload(my_df)

# Create dataset using a local file
with open("./data/examples.csv", "rb") as file_pointer:
    dataset_four = Dataset.create_from_file(filelike=file_pointer)

create_from_file()

classmethod create_from_file()

A blocking call that creates a new Dataset from a file. Returns when the dataset has been successfully uploaded and processed.

Warning: This function does not clean up it’s open files. If you pass a filelike, you are responsible for closing it. If you pass a file_path, this will create a file object from the file_path but will not close it.

Parameters

Parameter Type Description
file_path string, optional The path to the file. This will create a file object pointing to that file but will not close it.
filelike file, optional An open and readable file object.
categories list[string], optional An array of strings describing the intended use of the dataset. The current supported options are “TRAINING” and “PREDICTION”.
read_timeout Optional[int] The maximum number of seconds to wait for the server to respond indicating that the initial upload is complete
max_wait Optional[int] Time in seconds after which dataset creation is considered unsuccessful
use_cases list[UseCase] \| UseCase \| list[string] \| string, optional A list of UseCase objects, UseCase object, list of Use Case ids or a single Use Case id to add this new Dataset to. Must be a kwarg.

Returns

Returns Description
response A fully armed and operational Dataset

Return type: Dataset

create_from_in_memory_data()

classmethod create_from_in_memory_data()

A blocking call that creates a new Dataset from in-memory data. Returns when the dataset has been successfully uploaded and processed.

The data can be either a pandas DataFrame or a list of dictionaries with identical keys.

Parameters

Parameter Type Description
data_frame DataFrame, optional The data frame to upload
records list[dict], optional A list of dictionaries with identical keys to upload
categories list[string], optional An array of strings describing the intended use of the dataset. The current supported options are “TRAINING” and “PREDICTION”.
read_timeout Optional[int] The maximum number of seconds to wait for the server to respond indicating that the initial upload is complete
max_wait Optional[int] Time in seconds after which dataset creation is considered unsuccessful
fname string, optional The file name, “data.csv” by default
use_cases list[UseCase] \| UseCase \| list[string] \| string, optional A list of UseCase objects, UseCase object, list of Use Case IDs or a single Use Case ID to add this new dataset to. Must be a kwarg.

Returns

Returns Description
response The Dataset created from the uploaded data.

Return type: Dataset

Raises

Exception Description
InvalidUsageError If neither a DataFrame or list of records is passed.

create_from_url()

classmethod create_from_url()

A blocking call that creates a new Dataset from data stored at a url. Returns when the dataset has been successfully uploaded and processed.

Parameters

Parameter Type Description
url string The URL to use as the source of data for the dataset being created.
do_snapshot Optional[bool] If unset, uses the server default: True. If true, creates a snapshot dataset; if false, creates a remote dataset. Creating snapshots from non-file sources may be disabled by the permission, Disable AI Catalog Snapshots.
persist_data_after_ingestion Optional[bool] If unset, uses the server default: True. If true, will enforce saving all data (for download and sampling) and will allow a user to view extended data profile (which includes data statistics like min/max/median/mean, histogram, etc.). If false, will not enforce saving data. The data schema (feature names and types) still will be available. Specifying this parameter to false and doSnapshot to true will result in an error.
categories list[string], optional An array of strings describing the intended use of the dataset. The current supported options are “TRAINING” and “PREDICTION”.
sample_size dict, optional The size of data fetched during dataset registration. For example, to fetch the first 95 rows, the sample_size value would be:{‘type’: ‘rows’, ‘value’: 95}. Currently only ‘rows’ type is supported.
max_wait Optional[int] Time in seconds after which dataset creation is considered unsuccessful.
use_cases list[UseCase] \| UseCase \| list[string] \| string, optional A list of UseCase objects, UseCase object, list of Use Case IDs or a single Use Case ID to add this new dataset to. Must be a kwarg.

Returns

Returns Description
response The Dataset created from the uploaded data

Return type: Dataset

create_from_project()

classmethod create_from_project()

A blocking call that creates a new dataset from project data. Returns when the dataset has been successfully created.

Parameters

Parameter Type Description
project_id string The project to create the dataset from.
categories list[string], optional An array of strings describing the intended use of the dataset. The current supported options are “TRAINING” and “PREDICTION”.
max_wait Optional[int] Time in seconds after which dataset creation is considered unsuccessful.
use_cases list[UseCase] \| UseCase \| list[string] \| string, optional A list of UseCase objects, UseCase object, list of Use Case IDs or a single Use Case ID to add this new dataset to. Must be a kwarg.

Returns

Returns Description
response The dataset created from the project dataset.

Return type: Dataset

create_from_datastage()

classmethod create_from_datastage()

A blocking call that creates a new Dataset from data stored as a DataStage. Returns when the dataset has been successfully uploaded and processed.

Parameters

Parameter Type Description
datastage_id string The ID of the DataStage to use as the source of data for the dataset being created.
categories list[string], optional An array of strings describing the intended use of the dataset. The current supported options are “TRAINING” and “PREDICTION”.
max_wait Optional[int] Time in seconds after which dataset creation is considered unsuccessful.

Returns

Returns Description
response The Dataset created from the uploaded data

Return type: Dataset

create_from_data_source()

classmethod create_from_data_source()

A blocking call that creates a new Dataset from data stored at a DataSource. Returns when the dataset has been successfully uploaded and processed.

Added in version v2.22.

Parameters

Parameter Type Description
data_source_id string The ID of the DataSource to use as the source of data.
username string, optional The username for database authentication.
password string, optional The password (in cleartext) for database authentication. The password will be encrypted on the server side in scope of HTTP request and never saved or stored.
do_snapshot Optional[bool] If unset, uses the server default: True. If true, creates a snapshot dataset; if false, creates a remote dataset. Creating snapshots from non-file sources requires may be disabled by the permission, Disable AI Catalog Snapshots.
persist_data_after_ingestion Optional[bool] If unset, uses the server default: True. If true, will enforce saving all data (for download and sampling) and will allow a user to view extended data profile (which includes data statistics like min/max/median/mean, histogram, etc.). If false, will not enforce saving data. The data schema (feature names and types) still will be available. Specifying this parameter to false and doSnapshot to true will result in an error.
categories list[string], optional An array of strings describing the intended use of the dataset. The current supported options are “TRAINING” and “PREDICTION”.
credential_id string, optional The ID of the set of credentials to use instead of user and password. Note that with this change, username and password will become optional.
use_kerberos Optional[bool] If unset, uses the server default: False. If true, use kerberos authentication for database authentication.
credential_data dict, optional The credentials to authenticate with the database, to use instead of user/password or credential ID.
sample_size dict, optional The size of data fetched during dataset registration. For example, to fetch the first 95 rows, the sample_size value would be:{‘type’: ‘rows’, ‘value’: 95}. Currently only ‘rows’ type is supported.
max_wait Optional[int] Time in seconds after which project creation is considered unsuccessful.
use_cases list[UseCase] \| UseCase \| list[string] \| string, optional A list of UseCase objects, UseCase object, list of Use Case IDs or a single Use Case ID to add this new dataset to. Must be a kwarg.

Returns

Returns Description
response The Dataset created from the uploaded data

Return type: Dataset

create_from_query_generator()

classmethod create_from_query_generator()

A blocking call that creates a new Dataset from the query generator. Returns when the dataset has been successfully processed. If optional parameters are not specified the query is applied to the dataset_id and dataset_version_id stored in the query generator. If specified they will override the stored dataset_id/dataset_version_id, e.g., to prep a prediction dataset.

Parameters

Parameter Type Description
generator_id str The id of the query generator to use.
dataset_id Optional[str] The id of the dataset to apply the query to.
dataset_version_id Optional[str] The id of the dataset version to apply the query to. If not specified the latest version associated with dataset_id (if specified) is used.
max_wait int Optional; the maximum number of seconds to wait before giving up.
use_cases list[UseCase] \| UseCase \| list[string] \| string, optional A list of UseCase objects, UseCase object, list of Use Case IDs or a single Use Case ID to add this new dataset to. Must be a kwarg.

Returns

Returns Description
response The Dataset created from the query generator

Return type: Dataset

create_from_recipe()

classmethod create_from_recipe()

A blocking call that creates a new Dataset from the recipe. Returns when the dataset has been successfully uploaded and processed.

Added in version 3.6.

Returns

Returns Description
response The Dataset created from the uploaded data

Return type: Dataset

get()

classmethod get()

Get information about a dataset.

Parameters

Parameter Type Description
dataset_id string the ID of the dataset

Returns

Returns Description
dataset the queried dataset

Return type: Dataset

delete()

classmethod delete()

Soft deletes a dataset. You cannot get it or list it or do actions with it, except for un-deleting it.

Parameters

Parameter Type Description
dataset_id string The id of the dataset to mark for deletion

Return type: None

un_delete()

classmethod un_delete()

Un-deletes a previously deleted dataset. If the dataset was not deleted, nothing happens.

Parameters

Parameter Type Description
dataset_id string The id of the dataset to un-delete

Return type: None

list()

classmethod list()

List all datasets a user can view.

Parameters

Parameter Type Description
category string, optional Optional. If specified, only dataset versions that have the specified category will be included in the results. Categories identify the intended use of the dataset; supported categories are “TRAINING” and “PREDICTION”.
filter_failed Optional[bool] If unset, uses the server default: False. Whether datasets that failed during import should be excluded from the results. If True invalid datasets will be excluded.
order_by string, optional If unset, uses the server default: “-created”. Sorting order which will be applied to catalog list, valid options are:- “created” – ascending order by creation datetime;- “-created” – descending order by creation datetime.
use_cases Union[UseCase, List[UseCase], str, List[str]], optional Filter available datasets by a specific Use Case or Cases. Accepts either the entity or the ID. If set to [None], the method filters the project’s datasets by those not linked to a UseCase.

Returns

Returns Description
a list of datasets the user can view

Return type: list[Dataset]

iterate()

classmethod iterate()

Get an iterator for the requested datasets a user can view. This lazily retrieves results. It does not get the next page from the server until the current page is exhausted.

Parameters

Parameter Type Description
offset Optional[int] If set, this many results will be skipped
limit Optional[int] Specifies the size of each page retrieved from the server. If unset, uses the server default.
category string, optional Optional. If specified, only dataset versions that have the specified category will be included in the results. Categories identify the intended use of the dataset; supported categories are “TRAINING” and “PREDICTION”.
filter_failed Optional[bool] If unset, uses the server default: False. Whether datasets that failed during import should be excluded from the results. If True invalid datasets will be excluded.
order_by string, optional If unset, uses the server default: “-created”. Sorting order which will be applied to catalog list, valid options are:- “created” – ascending order by creation datetime;- “-created” – descending order by creation datetime.
use_cases Union[UseCase, List[UseCase], str, List[str]], optional Filter available datasets by a specific Use Case or Cases. Accepts either the entity or the ID. If set to [None], the method filters the project’s datasets by those not linked to a UseCase.

Yields

Yields Description
Dataset An iterator of the datasets the user can view.

Return type: Generator[TypeVar(TDataset, bound= Dataset), None, None]

update()

method update()

Updates the Dataset attributes in place with the latest information from the server.

Return type: None

modify()

method modify()

Modifies the Dataset name and/or categories. Updates the object in place.

Parameters

Parameter Type Description
name string, optional The new name of the dataset
categories list[string], optional A list of strings describing the intended use of the dataset. The supported options are “TRAINING” and “PREDICTION”. If any categories were previously specified for the dataset, they will be overwritten. If omitted or None, keep previous categories. To clear them specify []

Return type: None

modify_shared_roles()

method modify_shared_roles()

Grant access, remove access or update roles for users, groups and organizations who have access to this dataset. Up to 100 roles may be set in a single request.

Parameters

Parameter Type Description
roles List[CatalogSharedRoleRequest] An array of role requests to modify the roles for.
apply_grant_to_linked_objects bool If true for any users being granted access to the entity, grant the user read access to any linked objects such as DataSources and DataStores that may be used by this entity. Ignored if no such objects are relevant for the entity. This will not result in access being lowered for a user if the user already has higher access to linked objects than read access. However, if the target user does not have sharing permissions to the linked object, they will be given sharing access without lowering existing permissions. May result in an error if the user making the call does not have sufficient permissions to complete the grant.
operation str The name of the action being taken. The only supported operation is updateRoles.

Return type: None

Examples

Grant access to a user by name:

>>> from datarobot.enums import TARGET_SHARING_ROLE, SHARING_RECIPIENT_TYPE
>>> from datarobot.models.sharing import CatalogSharedRoleRequest
>>> from datarobot.models.dataset import Dataset

>>> dataset = Dataset.get("my-dataset-id")
>>> dataset.modify_shared_roles(
...     roles=[
...         CatalogSharedRoleRequest(
...             role=TARGET_SHARING_ROLE.EDITOR,
...             share_recipient_type=SHARING_RECIPIENT_TYPE.USER,
...             name="target_user_name",
...         )
...     ],
... )

Remove access to a group by ID:

>>> from datarobot.enums import TARGET_SHARING_ROLE, SHARING_RECIPIENT_TYPE
>>> from datarobot.models.sharing import CatalogSharedRoleRequest
>>> from datarobot.models.dataset import Dataset

>>> dataset = Dataset.get("my-dataset-id")
>>> dataset.modify_shared_roles(
...     roles=[
...         CatalogSharedRoleRequest(
...             role=TARGET_SHARING_ROLE.NO_ROLE,
...             share_recipient_type=SHARING_RECIPIENT_TYPE.GROUP,
...             id="group-id",
...         )
...     ],
... )

share()

method share()

Modify the ability of users to access this dataset

Parameters

Parameter Type Description
access_list list of SharingAccess The modifications to make.
apply_grant_to_linked_objects bool If true for any users being granted access to the dataset, grant the user read access to any linked objects such as DataSources and DataStores that may be used by this dataset. Ignored if no such objects are relevant for dataset, defaults to False.

Return type: None

Raises

Exception Description
datarobot.ClientError: If you do not have permission to share this dataset, if the user you’re sharing with doesn’t exist, if the same user appears multiple times in the access_list, or if these changes would leave the dataset without an owner.

Examples

Transfer access to the dataset from old_user@datarobot.com to new_user@datarobot.com

from datarobot.enums import SHARING_ROLE
from datarobot.models.dataset import Dataset
from datarobot.models.sharing import SharingAccess

new_access = SharingAccess(
    "new_user@datarobot.com",
    SHARING_ROLE.OWNER,
    can_share=True,
)
access_list = [
    SharingAccess(
        "old_user@datarobot.com",
        SHARING_ROLE.OWNER,
        can_share=True,
        can_use_data=True,
    ),
    new_access,
]

Dataset.get('my-dataset-id').share(access_list)

get_details()

method get_details()

Gets the details for this Dataset

Return type: DatasetDetails

get_all_features()

method get_all_features()

Get a list of all the features for this dataset.

Parameters

Parameter Type Description
order_by string, optional If unset, uses the server default: ‘name’. How the features should be ordered. Can be ‘name’ or ‘featureType’.

Return type: list[DatasetFeature]

get_shared_roles()

method get_shared_roles()

List roles for users, groups, and organizations who have access to this dataset.

Parameters

Parameter Type Description
id Optional[str] Only return the access control information for a organization, group or user with this ID.
name Optional[str] Only return the access control information for a organization, group or user with this name.
share_recipient_type Optional[SHARING_RECIPIENT_TYPE] Only returns results with the given recipient type.

Return type: List[CatalogSharedRole]

iterate_all_features()

method iterate_all_features()

Get an iterator for the requested features of a dataset. This lazily retrieves results. It does not get the next page from the server until the current page is exhausted.

Parameters

Parameter Type Description
offset Optional[int] If set, this many results will be skipped.
limit Optional[int] Specifies the size of each page retrieved from the server. If unset, uses the server default.
order_by string, optional If unset, uses the server default: ‘name’. How the features should be ordered. Can be ‘name’ or ‘featureType’.

Yields

Yields Description
DatasetFeature

Return type: Generator[DatasetFeature, None, None]

get_featurelists()

method get_featurelists()

Get DatasetFeaturelists created on this Dataset

Returns

Returns Description
feature_lists

Return type: list[DatasetFeaturelist]

create_featurelist()

method create_featurelist()

Create a new dataset featurelist

Parameters

Parameter Type Description
name str the name of the modeling featurelist to create. Names must be unique within the dataset, or the server will return an error.
features List[str] the names of the features to include in the dataset featurelist. Each feature must be a dataset feature.

Returns

Returns Description
featurelist the newly created featurelist

Return type: DatasetFeaturelist

Examples

dataset = Dataset.get('1234deadbeeffeeddead4321')
dataset_features = dataset.get_all_features()
selected_features = [feat.name for feat in dataset_features][:5]  # select first five
new_flist = dataset.create_featurelist('Simple Features', selected_features)

get_file()

method get_file()

Retrieves all the originally uploaded data in CSV form. Writes it to either the file or a filelike object that can write bytes.

Only one of file_path or filelike can be provided and it must be provided as a keyword argument (i.e., file_path=’path-to-write-to’). If a file-like object is provided, the user is responsible for closing it when they are done.

The user must also have permission to download data.

Parameters

Parameter Type Description
file_path string, optional The destination to write the file to.
filelike file, optional A file-like object to write to. The object must be able to write bytes. The user is responsible for closing the object

Return type: None

get_as_dataframe()

method get_as_dataframe()

Retrieves all the originally uploaded data in a pandas DataFrame.

Added in version v3.0.

Parameters

Parameter Type Description
low_memory Optional[bool] If True, use local files to reduce memory usage which will be slower.

Return type: pd.DataFrame

get_projects()

method get_projects()

Retrieves the Dataset’s projects as ProjectLocation named tuples.

Returns

Returns Description
locations

Return type: list[ProjectLocation]

get_raw_sample_data()

method get_raw_sample_data()

Retrieves the raw sample data for the dataset as a pandas DataFrame. The raw sample dataset is a subset of the full dataset.

Added in version v3.10.

Returns

Returns Description
A DataFrame with the dataset’s raw sample data.

Return type: pd.DataFrame

create_project()

method create_project()

Create a datarobot.models.Project from this dataset

Parameters

Parameter Type Description
project_name string, optional The name of the project to be created. If not specified, will be “Untitled Project” for database connections, otherwise the project name will be based on the file used.
user string, optional The username for database authentication.
password string, optional The password (in cleartext) for database authentication. The password will be encrypted on the server side in scope of HTTP request and never saved or stored
credential_id string, optional The ID of the set of credentials to use instead of user and password.
use_kerberos Optional[bool] Server default is False. If true, use kerberos authentication for database authentication.
credential_data dict, optional The credentials to authenticate with the database, to use instead of user/password or credential ID.
use_cases list[UseCase] \| UseCase \| list[string] \| string, optional A list of UseCase objects, UseCase object, list of Use Case ids or a single Use Case id to add this new Dataset to. Must be a kwarg.

Return type: Project

create_version_from_file()

classmethod create_version_from_file()

A blocking call that creates a new Dataset version from a file. Returns when the new dataset version has been successfully uploaded and processed.

Warning: This function does not clean up it’s open files. If you pass a filelike, you are responsible for closing it. If you pass a file_path, this will create a file object from the file_path but will not close it.

Added in version v2.23.

Parameters

Parameter Type Description
dataset_id string The ID of the dataset for which new version to be created
file_path string, optional The path to the file. This will create a file object pointing to that file but will not close it.
filelike file, optional An open and readable file object.
categories list[string], optional An array of strings describing the intended use of the dataset. The current supported options are “TRAINING” and “PREDICTION”.
read_timeout Optional[int] The maximum number of seconds to wait for the server to respond indicating that the initial upload is complete
max_wait Optional[int] Time in seconds after which project creation is considered unsuccessful

Returns

Returns Description
response A fully armed and operational Dataset version

Return type: Dataset

create_version_from_in_memory_data()

classmethod create_version_from_in_memory_data()

A blocking call that creates a new Dataset version for a dataset from in-memory data. Returns when the dataset has been successfully uploaded and processed.

The data can be either a pandas DataFrame or a list of dictionaries with identical keys.

Added in version v2.23.

Parameters

Parameter Type Description
dataset_id string The ID of the dataset for which new version to be created
data_frame DataFrame, optional The data frame to upload
records list[dict], optional A list of dictionaries with identical keys to upload
categories list[string], optional An array of strings describing the intended use of the dataset. The current supported options are “TRAINING” and “PREDICTION”.
read_timeout Optional[int] The maximum number of seconds to wait for the server to respond indicating that the initial upload is complete
max_wait Optional[int] Time in seconds after which project creation is considered unsuccessful

Returns

Returns Description
response The Dataset version created from the uploaded data

Return type: Dataset

Raises

Exception Description
InvalidUsageError If neither a DataFrame or list of records is passed.

create_version_from_url()

classmethod create_version_from_url()

A blocking call that creates a new Dataset from data stored at a url for a given dataset. Returns when the dataset has been successfully uploaded and processed.

Added in version v2.23.

Parameters

Parameter Type Description
dataset_id string The ID of the dataset for which new version to be created
url string The URL to use as the source of data for the dataset being created.
categories list[string], optional An array of strings describing the intended use of the dataset. The current supported options are “TRAINING” and “PREDICTION”.
max_wait Optional[int] Time in seconds after which project creation is considered unsuccessful

Returns

Returns Description
response The Dataset version created from the uploaded data

Return type: Dataset

create_version_from_datastage()

classmethod create_version_from_datastage()

A blocking call that creates a new Dataset from data stored as a DataStage for a given dataset. Returns when the dataset has been successfully uploaded and processed.

Parameters

Parameter Type Description
dataset_id string The ID of the dataset for which new version to be created
datastage_id string The ID of the DataStage to use as the source of data for the dataset being created.
categories list[string], optional An array of strings describing the intended use of the dataset. The current supported options are “TRAINING” and “PREDICTION”.
max_wait Optional[int] Time in seconds after which project creation is considered unsuccessful

Returns

Returns Description
response The Dataset version created from the uploaded data

Return type: Dataset

create_version_from_data_source()

classmethod create_version_from_data_source()

A blocking call that creates a new Dataset from data stored at a DataSource. Returns when the dataset has been successfully uploaded and processed.

Added in version v2.23.

Parameters

Parameter Type Description
dataset_id string The ID of the dataset for which new version to be created
data_source_id string The ID of the DataSource to use as the source of data.
username string, optional The username for database authentication.
password string, optional The password (in cleartext) for database authentication. The password will be encrypted on the server side in scope of HTTP request and never saved or stored.
categories list[string], optional An array of strings describing the intended use of the dataset. The current supported options are “TRAINING” and “PREDICTION”.
credential_id string, optional The ID of the set of credentials to use instead of user and password. Note that with this change, username and password will become optional.
use_kerberos Optional[bool] If unset, uses the server default: False. If true, use kerberos authentication for database authentication.
credential_data dict, optional The credentials to authenticate with the database, to use instead of user/password or credential ID.
max_wait Optional[int] Time in seconds after which project creation is considered unsuccessful

Returns

Returns Description
response The Dataset version created from the uploaded data

Return type: Dataset

create_version_from_recipe()

classmethod create_version_from_recipe()

A blocking call that creates a new Dataset version from Recipe. Returns when the dataset has been successfully uploaded and processed.

Added in version v3.8.

Parameters

Parameter Type Description
dataset_id string The ID of the dataset for which a new version will be created.
recipe Recipe The Recipe to use to create a new dataset version.
credential Credential, optional The credentials to authenticate with the database.
credential_id string, optional The ID of the set of credentials to use instead of Credential object.
use_kerberos Optional[bool] If unset, uses the server default: False. If true, use kerberos authentication for database authentication.
max_wait Optional[int] Time in seconds after which project creation is considered unsuccessful.

Returns

Returns Description
response The Dataset version created from the uploaded data

Return type: Dataset

from_data()

classmethod from_data()

Instantiate an object of this class using a dict.

Parameters

Parameter Type Description
data dict Correctly snake_cased keys and their values.

Return type: TypeVar(T, bound= APIObject)

from_server_data()

classmethod from_server_data()

Instantiate an object of this class using the data directly from the server, meaning that the keys may have the wrong camel casing

Parameters

Parameter Type Description
data dict The directly translated dict of JSON from the server. No casing fixes have taken place
keep_attrs iterable List, set or tuple of the dotted namespace notations for attributes to keep within the object structure even if their values are None

Return type: TypeVar(T, bound= APIObject)

open_in_browser()

method open_in_browser()

Opens class’ relevant web browser location. If default browser is not available the URL is logged.

Note: If text-mode browsers are used, the calling process will block until the user exits the browser.

Return type: None

DatasetDetails

class datarobot.DatasetDetails

Represents a detailed view of a Dataset. The to_dataset method creates a Dataset from this details view.

Variables

Attribute Type Description
dataset_id string The ID of this dataset
name string The name of this dataset in the catalog
is_latest_version bool Whether this dataset version is the latest version of this dataset
version_id string The object ID of the catalog_version the dataset belongs to
categories list(string) An array of strings describing the intended use of the dataset. The supported options are “TRAINING” and “PREDICTION”.
created_at string The date when the dataset was created
created_by string Username of the user who created the dataset
is_snapshot bool Whether the dataset version is an immutable snapshot of data which has previously been retrieved and saved to Data_robot
data_persisted Optional[bool] If true, user is allowed to view extended data profile (which includes data statistics like min/max/median/mean, histogram, etc.) and download data. If false, download is not allowed and only the data schema (feature names and types) will be available.
is_data_engine_eligible bool Whether this dataset can be a data source of a data engine query.
processing_state string Current ingestion process state of the dataset
row_count Optional[int] The number of rows in the dataset.
size Optional[int] The size of the dataset as a CSV in bytes.
data_engine_query_id string, optional ID of the source data engine query
data_source_id string, optional ID of the datasource used as the source of the dataset
data_source_type string the type of the datasource that was used as the source of the dataset
description string, optional the description of the dataset
eda1_modification_date string, optional the ISO 8601 formatted date and time when the EDA1 for the dataset was updated
eda1_modifier_full_name string, optional the user who was the last to update EDA1 for the dataset
error string details of exception raised during ingestion process, if any
feature_count Optional[int] total number of features in the dataset
feature_count_by_type list[FeatureTypeCount] number of features in the dataset grouped by feature type
last_modification_date string the ISO 8601 formatted date and time when the dataset was last modified
last_modifier_full_name string full name of user who was the last to modify the dataset
tags list[string] list of tags attached to the item
uri string the uri to datasource like:- ‘file_name.csv’ - ‘jdbc:DATA_SOURCE_GIVEN_NAME/SCHEMA.TABLE_NAME’ - ‘jdbc:DATA_SOURCE_GIVEN_NAME/’ - for query based datasources - ‘https://s3.amazonaws.com/my_data/my_dataset.csv’ - etc.
sample_size dict, optional The size of data fetched during dataset registration. For example, to fetch the first 95 rows, the sample_size value is {‘type’: ‘rows’, ‘value’: 95}. Currently only ‘rows’ type is supported.

get()

classmethod get()

Get details for a Dataset from the server

Parameters

Parameter Type Description
dataset_id str The id for the Dataset from which to get details

Return type: DatasetDetails

to_dataset()

method to_dataset()

Build a Dataset object from the information in this object

Return type: Dataset

ProjectLocation

class datarobot.models.dataset.ProjectLocation

ProjectLocation(url, id)

id

method id

Alias for field number 1

url

method url

Alias for field number 0

Secondary datasets

SecondaryDataset

class datarobot.helpers.feature_discovery.SecondaryDataset

A secondary dataset to be used for feature discovery

Added in version v2.25.

Variables

Attribute Type Description
identifier str Alias of the dataset (used directly as part of the generated feature names)
catalog_id str Identifier of the catalog item
catalog_version_id str Identifier of the catalog item version
snapshot_policy Optional[str] Policy to use while creating a project or making predictions. If omitted, by default endpoint will use ‘latest’. Must be one of the following values:‘specified’: Use specific snapshot specified by catalogVersionId ‘latest’: Use latest snapshot from the same catalog item ‘dynamic’: Get data from the source (only applicable for JDBC datasets)

Examples

import datarobot as dr
dataset_definition = dr.SecondaryDataset(
    identifier='profile',
    catalog_id='5ec4aec1f072bc028e3471ae',
    catalog_version_id='5ec4aec2f072bc028e3471b1',
)

Secondary dataset configurations

SecondaryDatasetConfigurations

class datarobot.models.SecondaryDatasetConfigurations

Create secondary dataset configurations for a given project

Added in version v2.20.

Variables

Attribute Type Description
id str Id of this secondary dataset configuration
project_id str Id of the associated project.
config list of DatasetConfiguration (Deprecated in version v2.23) List of secondary dataset configurations
secondary_datasets list of SecondaryDataset (new in v2.23) List of secondary datasets (secondaryDataset)
name str Verbose name of the SecondaryDatasetConfig. null if it wasn’t specified.
created datetime.datetime DR-formatted datetime. null for legacy (before DR 6.0) db records.
creator_user_id str Id of the user created this config.
creator_full_name str fullname or email of the user created this config.
featurelist_id Optional[str] Id of the feature list. null if it wasn’t specified.
credential_ids Optional[list of DatasetsCredentials] credentials used by the secondary datasets if the datasets used in the configuration are from datasource
is_default Optional[bool] Boolean flag if default config created during feature discovery aim
project_version Optional[str] Version of project when its created (Release version)

create()

classmethod create()

create secondary dataset configurations

Added in version v2.20.

Parameters

Parameter Type Description
project_id str id of the associated project.
secondary_datasets list of SecondaryDataset (New in version v2.23) list of secondary datasets used by the configuration each element is a datarobot.helpers.feature_discovery.SecondaryDataset
name str (New in version v2.23) Name of the secondary datasets configuration
featurelist_id str, or None (New in version v2.23) Id of the featurelist

Return type: an instance of SecondaryDatasetConfigurations

Raises

Exception Description
ClientError raised if incorrect configuration parameters are provided

Examples

   profile_secondary_dataset = dr.SecondaryDataset(
       identifier='profile',
       catalog_id='5ec4aec1f072bc028e3471ae',
       catalog_version_id='5ec4aec2f072bc028e3471b1',
       snapshot_policy='latest'
   )

   transaction_secondary_dataset = dr.SecondaryDataset(
       identifier='transaction',
       catalog_id='5ec4aec268f0f30289a03901',
       catalog_version_id='5ec4aec268f0f30289a03900',
       snapshot_policy='latest'
   )

   secondary_datasets = [profile_secondary_dataset, transaction_secondary_dataset]
   new_secondary_dataset_config = dr.SecondaryDatasetConfigurations.create(
       project_id=project.id,
       name='My config',
       secondary_datasets=secondary_datasets
   )

>>> new_secondary_dataset_config.id
'5fd1e86c589238a4e635e93d'

delete()

method delete()

Removes the Secondary datasets configuration

Added in version v2.21.

Raises

Exception Description
ClientError Raised if an invalid or already deleted secondary dataset config id is provided

Examples

# Deleting with a valid secondary_dataset_config id
status_code = dr.SecondaryDatasetConfigurations.delete(some_config_id)
status_code
>>> 204

Return type: None

get()

method get()

Retrieve a single secondary dataset configuration for a given id

Added in version v2.21.

Returns

Returns Description
secondary_dataset_configurations The requested secondary dataset configurations

Return type: SecondaryDatasetConfigurations

Examples

config_id = '5fd1e86c589238a4e635e93d'
secondary_dataset_config = dr.SecondaryDatasetConfigurations(id=config_id).get()
>>> secondary_dataset_config
{
     'created': datetime.datetime(2020, 12, 9, 6, 16, 22, tzinfo=tzutc()),
     'creator_full_name': u'abc@datarobot.com',
     'creator_user_id': u'asdf4af1gf4bdsd2fba1de0a',
     'credential_ids': None,
     'featurelist_id': None,
     'id': u'5fd1e86c589238a4e635e93d',
     'is_default': True,
     'name': u'My config',
     'project_id': u'5fd06afce2456ec1e9d20457',
     'project_version': None,
     'secondary_datasets': [
            {
                'snapshot_policy': u'latest',
                'identifier': u'profile',
                'catalog_version_id': u'5fd06b4af24c641b68e4d88f',
                'catalog_id': u'5fd06b4af24c641b68e4d88e'
            },
            {
                'snapshot_policy': u'dynamic',
                'identifier': u'transaction',
                'catalog_version_id': u'5fd1e86c589238a4e635e98e',
                'catalog_id': u'5fd1e86c589238a4e635e98d'
            }
     ]
}

list()

classmethod list()

Returns list of secondary dataset configurations.

Added in version v2.23.

Parameters

Parameter Type Description
project_id str The Id of project
featurelist_id Optional[str] Id of the feature list to filter the secondary datasets configurations

Returns

Returns Description
secondary_dataset_configurations The requested list of secondary dataset configurations for a given project

Return type: list of SecondaryDatasetConfigurations

Examples

pid = '5fd06afce2456ec1e9d20457'
secondary_dataset_configs = dr.SecondaryDatasetConfigurations.list(pid)
>>> secondary_dataset_configs[0]
    {
         'created': datetime.datetime(2020, 12, 9, 6, 16, 22, tzinfo=tzutc()),
         'creator_full_name': u'abc@datarobot.com',
         'creator_user_id': u'asdf4af1gf4bdsd2fba1de0a',
         'credential_ids': None,
         'featurelist_id': None,
         'id': u'5fd1e86c589238a4e635e93d',
         'is_default': True,
         'name': u'My config',
         'project_id': u'5fd06afce2456ec1e9d20457',
         'project_version': None,
         'secondary_datasets': [
                {
                    'snapshot_policy': u'latest',
                    'identifier': u'profile',
                    'catalog_version_id': u'5fd06b4af24c641b68e4d88f',
                    'catalog_id': u'5fd06b4af24c641b68e4d88e'
                },
                {
                    'snapshot_policy': u'dynamic',
                    'identifier': u'transaction',
                    'catalog_version_id': u'5fd1e86c589238a4e635e98e',
                    'catalog_id': u'5fd1e86c589238a4e635e98d'
                }
         ]
    }

Data engine query generator

DataEngineQueryGenerator

class datarobot.DataEngineQueryGenerator

DataEngineQueryGenerator is used to set up time series data prep.

Added in version v2.27.

Variables

Attribute Type Description
id str ID of the query generator
query str Text of the generated Spark SQL query
datasets list(QueryGeneratorDataset) Datasets associated with the query generator
generator_settings QueryGeneratorSettings The settings used to define the query
generator_type str “TimeSeries” is the only supported type

create()

classmethod create()

Creates a query generator entity.

Added in version v2.27.

Parameters

Parameter Type Description
generator_type str Type of data engine query generator
datasets List[QueryGeneratorDataset] Source datasets in the Data Engine workspace.
generator_settings QueryGeneratorSettings Data engine generator settings of the given generator_type.

Returns

Returns Description
query_generator The created generator

Return type: DataEngineQueryGenerator

Examples

import datarobot as dr
from datarobot.models.data_engine_query_generator import (
   QueryGeneratorDataset,
   QueryGeneratorSettings,
)
dataset = QueryGeneratorDataset(
   alias='My_Awesome_Dataset_csv',
   dataset_id='61093144cabd630828bca321',
   dataset_version_id=1,
)
settings = QueryGeneratorSettings(
   datetime_partition_column='date',
   time_unit='DAY',
   time_step=1,
   default_numeric_aggregation_method='sum',
   default_categorical_aggregation_method='mostFrequent',
)
g = dr.DataEngineQueryGenerator.create(
   generator_type='TimeSeries',
   datasets=[dataset],
   generator_settings=settings,
)
g.id
>>>'54e639a18bd88f08078ca831'
g.generator_type
>>>'TimeSeries'

get()

classmethod get()

Gets information about a query generator.

Parameters

Parameter Type Description
generator_id str The identifier of the query generator you want to load.

Returns

Returns Description
query_generator The queried generator

Return type: DataEngineQueryGenerator

Examples

import datarobot as dr
g = dr.DataEngineQueryGenerator.get(generator_id='54e639a18bd88f08078ca831')
g.id
>>>'54e639a18bd88f08078ca831'
g.generator_type
>>>'TimeSeries'

create_dataset()

method create_dataset()

A blocking call that creates a new Dataset from the query generator. Returns when the dataset has been successfully processed. If optional parameters are not specified the query is applied to the dataset_id and dataset_version_id stored in the query generator. If specified they will override the stored dataset_id/dataset_version_id, i.e., to prep a prediction dataset.

Parameters

Parameter Type Description
dataset_id Optional[str] The ID of the unprepped dataset to apply the query to
dataset_version_id Optional[str] The version_id of the unprepped dataset to apply the query to

Returns

Returns Description
response The Dataset created from the query generator

Return type: Dataset

prepare_prediction_dataset_from_catalog()

method prepare_prediction_dataset_from_catalog()

Apply time series data prep to a catalog dataset and upload it to the project as a PredictionDataset.

Added in version v3.1.

Parameters

Parameter Type Description
project_id str The id of the project to which you upload the prediction dataset.
dataset_id str The identifier of the dataset.
dataset_version_id Optional[str] The version id of the dataset to use.
max_wait Optional[int] Optional, the maximum number of seconds to wait before giving up.
relax_known_in_advance_features_check Optional[bool] For time series projects only. If True, missing values in the known in advance features are allowed in the forecast window at the prediction time. If omitted or False, missing values are not allowed.

Returns

Returns Description
dataset The newly uploaded dataset.

Return type: PredictionDataset

prepare_prediction_dataset()

method prepare_prediction_dataset()

Apply time series data prep and upload the PredictionDataset to the project.

Added in version v3.1.

Parameters

Parameter Type Description
sourcedata str, file or pandas.DataFrame Data to be used for predictions. If it is a string, it can be either a path to a local file, or raw file content. If using a file on disk, the filename must consist of ASCII characters only.
project_id str The id of the project to which you upload the prediction dataset.
max_wait Optional[int] The maximum number of seconds to wait for the uploaded dataset to be processed before raising an error.
relax_known_in_advance_features_check Optional[bool] For time series projects only. If True, missing values in the known in advance features are allowed in the forecast window at the prediction time. If omitted or False, missing values are not allowed.

Returns

Returns Description
dataset The newly uploaded dataset.

Return type: PredictionDataset

Raises

Exception Description
InputNotUnderstoodError Raised if sourcedata isn’t one of supported types.
AsyncFailureError Raised if polling for the status of an async process resulted in a response with an unsupported status code.
AsyncProcessUnsuccessfulError Raised if project creation was unsuccessful (i.e., the server reported an error in uploading the dataset).
AsyncTimeoutError Raised if processing the uploaded dataset took more time than specified by the max_wait parameter.

Sharing access

SharingAccess

class datarobot.SharingAccess

Represents metadata about whom a entity (e.g., a data store) has been shared with

Added in version v2.14.

Currently DataStores, DataSources, Datasets, Projects (new in version v2.15) and CalendarFiles (new in version 2.15) can be shared.

This class can represent either access that has already been granted, or be used to grant access to additional users.

Variables

Attribute Type Description
username str a particular user
role str or None if a string, represents a particular level of access and should be one of datarobot.enums.SHARING_ROLE. For more information on the specific access levels, see the sharing documentation. If None, can be passed to a share function to revoke access for a specific user.
can_share bool or None if a bool, indicates whether this user is permitted to further share. When False, the user has access to the entity, but can only revoke their own access but not modify any user’s access role. When True, the user can share with any other user at a access role up to their own. May be None if the SharingAccess was not retrieved from the DataRobot server but intended to be passed into a share function; this will be equivalent to passing True.
can_use_data bool or None if a bool, indicates whether this user should be able to view, download and process data (use to create projects, predictions, etc). For OWNER can_use_data is always True. If role is empty canUseData is ignored.
user_id str or None the ID of the user

Sharing role

SharingRole

class datarobot.models.sharing.SharingRole

Represents metadata about a user who has been granted access to an entity. At least one of id or username must be set.

Variables

Attribute Type Description
id str or None The ID of the user.
role str Represents a particular level of access. Should be one of datarobot.enums.SHARING_ROLE.
share_recipient_type SHARING_RECIPIENT_TYPE The type of user for the object of the method. Can be user or organization.
user_full_name str or None The full name of the user.
username str or None The username (usually the email) of the user.
can_share bool or None Indicates whether this user is permitted to share with other users. When False, the user has access to the entity, but can only revoke their own access. They cannot not modify any user’s access role. When True, the user can share with any other user at an access role up to their own.