Batch predictions¶

The batch prediction API provides a way to score large datasets using flexible options for intake and output on the Prediction Servers you have already deployed.

The main features are:

• Flexible options for intake and output.
• Stream local files and start scoring while still uploading and simultaneously downloading the results.
• Score large datasets from and to S3.
• Connect to your database using JDBC with bidirectional streaming of scoring data and results.
• Intake and output options can be mixed and do not need to match. So scoring from a JDBC source to an S3 target is also an option.
• Protection against overloading your prediction servers with the option to control the concurrency level for scoring.
• Prediction explanations can be included (with the option to add thresholds).
• Passthrough columns are supported to correlate scored data with source data.
• You can include prediction warnings in the output.

To interact with batch predictions, see the BatchPredictionJob class.

Make batch predictions with a deployment¶

DataRobot provides a utility function to make batch predictions using a deployment: Deployment.predict_batch.

import datarobot as dr

deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0')
# To note: `source` can be a file path, a file or a pandas DataFrame
prediction_results_as_dataframe = deployment.predict_batch(
    source="./my_local_file.csv",
)

Scoring local CSV files¶

DataRobot provides a utility function for scoring to and from local CSV files: BatchPredictionJob.score_to_file. The first parameter can be either:

• A path to a CSV dataset
• A file-like object
• A Pandas DataFrame

For larger datasets, you should avoid using a DataFrame, as it loads the entire dataset into memory. The other options do not.

import datarobot as dr

deployment_id = '5dc5b1015e6e762a6241f9aa'

dr.BatchPredictionJob.score_to_file(
    deployment_id,
    './data_to_predict.csv',
    './predicted.csv',
)

The input file is streamed to DataRobot’s API and scoring starts immediately. As soon as results start coming in, they start to be downloaded. The entire call is blocked until the file has been scored.

Scoring from and to S3¶

DataRobot provides a small utility function for scoring to and from CSV files hosted on S3: BatchPredictionJob.score_s3. This requires that the intake and output buckets share the same credentials (see Credentials and Credential.create_s3) or that their access policy is set to public:

Note that the S3 output functionality has a limit of 100 GB.

import datarobot as dr

deployment_id = '5dc5b1015e6e762a6241f9aa'

cred = dr.Credential.get('5a8ac9ab07a57a0001be501f')

job = dr.BatchPredictionJob.score_s3(
    deployment=deployment_id,
    source_url='s3://mybucket/data_to_predict.csv',
    destination_url='s3://mybucket/predicted.csv',
    credential=cred,
)

Scoring from and to Azure Cloud Storage¶

DataRobot provides the same support for Azure through the utility function BatchPredictionJob.score_azure. This requires that you add an Azure connection string to the DataRobot credentials store. (see Credentials and Credential.create_azure)

import datarobot as dr

deployment_id = '5dc5b1015e6e762a6241f9aa'

cred = dr.Credential.get('5a8ac9ab07a57a0001be501f')

job = dr.BatchPredictionJob.score_azure(
    deployment=deployment_id,
    source_url='https://mybucket.blob.core.windows.net/bucket/data_to_predict.csv',
    destination_url='https://mybucket.blob.core.windows.net/results/predicted.csv',
    credential=cred,
)

Scoring from and to Google Cloud Platform¶

DataRobot provides the same support for GCP through the utility function BatchPredictionJob.score_gcp. It requires you to add a GCP connection string to the DataRobot credentials store. (See Credentials and Credential.create_gcp.)

import datarobot as dr

deployment_id = '5dc5b1015e6e762a6241f9aa'

cred = dr.Credential.get('5a8ac9ab07a57a0001be501f')

job = dr.BatchPredictionJob.score_gcp(
    deployment=deployment_id,
    source_url='gs:/bucket/data_to_predict.csv',
    destination_url='gs://results/predicted.csv',
    credential=cred,
)

Manually configure a batch prediction job¶

If you can’t use any of the utilities above, you are also free to manually configure your job. This requires configuring an intake and output option using the intake_settings and output_settings parameters on BatchPredictionJob.score. Credentials may be created with Credentials API.

import datarobot as dr

deployment_id = '5dc5b1015e6e762a6241f9aa'

dr.BatchPredictionJob.score(
    deployment_id,
    intake_settings={
        'type': 's3',
        'url': 's3://public-bucket/data_to_predict.csv',
        'credential_id': '5a8ac9ab07a57a0001be501f',
    },
    output_settings={
        'type': 'localFile',
        'path': './predicted.csv',
    },
)

Supported intake types¶

The type field selects the intake adapter. Supported values are localFile, s3, azure, gcp, dataset, jdbc, snowflake, synapse, bigquery, and datasphere. Intake and output types can be mixed (for example, JDBC intake to S3 output).

The following sections describe configuration parameters for each intake type:

Local file intake¶

Set type to localFile and pass scoring data with file—a file-like object, a string path to a CSV file, or a pandas.DataFrame:

intake_settings={
    'type': 'localFile',
    'file': './data_to_predict.csv',
}

S3 CSV intake¶

S3 CSV intake requires you to pass an S3 URL to the CSV file to be scored in the url parameter. Optional parameters:

• credential_id : str, ID of stored AWS credentials (see Credential API).
• endpoint_url : str, a non-default S3 endpoint URL. Omit to use the default AWS endpoint.

intake_settings={
    'type': 's3',
    'url': 's3://public-bucket/data_to_predict.csv',
}

If the bucket is not publicly accessible, you can supply AWS credentials using the following parameters:

• aws_access_key_id
• aws_secret_access_key
• aws_session_token

Save it to the Credential API:

import datarobot as dr

# get to make sure it exists
credential_id = '5a8ac9ab07a57a0001be501f'
cred = dr.Credential.get(credential_id)

intake_settings={
    'type': 's3',
    'url': 's3://private-bucket/data_to_predict.csv',
    'credential_id': cred.credential_id,
}

JDBC intake¶

JDBC intake requires you to create a DataStore and Credential for your database. Identify the data source with either a SQL query or a combination of table, schema, and/or catalog:

• data_store_id : str, the ID of the external data store connected to the JDBC data source.
• query : str (optional if table, schema, and/or catalog is specified), a SELECT statement for the data to predict.
• table : str (optional if query is specified), the database table name.
• schema : str (optional if query is specified), the database schema name.
• catalog : str (optional if query is specified), the database catalog name (new in v2.22).
• fetch_size : Optional[int], row batch size when reading from the database. Adjust to balance throughput and memory usage.
• credential_id : Optional[str], ID of credentials with read access (see Credential).

# get to make sure it exists
datastore_id = '5a8ac9ab07a57a0001be5010'
data_store = dr.DataStore.get(datastore_id)

credential_id = '5a8ac9ab07a57a0001be501f'
cred = dr.Credential.get(credential_id)

intake_settings = {
    'type': 'jdbc',
    'table': 'table_name',
    'schema': 'public', # optional, if supported by database
    'catalog': 'master', # optional, if supported by database
    'data_store_id': data_store.id,
    'credential_id': cred.credential_id,
}

Azure blob storage intake¶

Azure intake uses the same parameters as S3 intake, with an Azure blob storage URL in url. See BatchPredictionJob.score_azure for a utility method, or configure manually:

intake_settings={
    'type': 'azure',
    'url': 'https://storage_account.blob.core.windows.net/container/data_to_predict.csv',
    'credential_id': '5a8ac9ab07a57a0001be501f',
}

Google Cloud Storage intake¶

GCP intake uses the same parameters as S3 intake, with a GCS URL in url. See BatchPredictionJob.score_gcp for a utility method, or configure manually:

intake_settings={
    'type': 'gcp',
    'url': 'gs://bucket/data_to_predict.csv',
    'credential_id': '5a8ac9ab07a57a0001be501f',
}

Snowflake and Synapse intake¶

For Snowflake and Synapse data sources, set type to snowflake or synapse and use the same parameters as JDBC intake.

BigQuery intake¶

BigQuery intake requires you to create a GCS Credential for your database:

# get to make sure it exists
credential_id = '5a8ac9ab07a57a0001be501f'
cred = dr.Credential.get(credential_id)

intake_settings = {
    'type': 'bigquery',
    'dataset': 'dataset_name',
    'table': 'table_or_view_name',
    'bucket': 'bucket_in_gcs',
    'credential_id': cred.credential_id,
}

AI Catalog intake¶

Set type to dataset and pass the scoring data as a dr.Dataset object in the dataset parameter. You must set both type and dataset; passing a dataset ID string alone is not supported.

Create a Dataset and retrieve it with dr.Dataset.get():

# get to make sure it exists
dataset_id = '5a8ac9ab07a57a0001be501f'
dataset = dr.Dataset.get(dataset_id)

intake_settings={
    'type': 'dataset',
    'dataset': dataset
}

Or, if you want a version_id other than the latest, supply your own.

# get to make sure it exists
dataset_id = '5a8ac9ab07a57a0001be501f'
dataset = dr.Dataset.get(dataset_id)

intake_settings={
    'type': 'dataset',
    'dataset': dataset,
    'dataset_version_id': 'another_version_id'
}

Datasphere intake¶

Datasphere intake requires you to create a DataStore and Credential for your database. Set type to datasphere and pass the following parameters:

# get to make sure it exists
datastore_id = '5a8ac9ab07a57a0001be5011'
data_store = dr.DataStore.get(datastore_id)

credential_id = '5a8ac9ab07a57a0001be501f'
cred = dr.Credential.get(credential_id)

intake_settings = {
    'type': 'datasphere',
    'table': 'table_name',
    'schema': 'DATASPHERE_SPACE_NAME',
    'data_store_id': data_store.id,
    'credential_id': cred.credential_id,
}

DSS (training data) intake¶

For scoring subsets of training data with a leaderboard model, use the dss intake type. You must also set timeseries_settings with type set to training.

• project_id : str, the project to fetch training data from. Access to the project is required.
• partition : str, subset of training data to score, one of datarobot.enums.TrainingDataSubsets.

intake_settings = {
    'type': 'dss',
    'project_id': '5a8ac9ab07a57a0001be5010',
    'partition': 'holdout',
}

Supported output types¶

The type field selects the output adapter. Supported values are localFile, s3, azure, gcp, jdbc, snowflake, synapse, bigquery, and datasphere.

The sections below describe configuration parameters for each output type.

Local file output¶

Set type to localFile. The optional path parameter controls how results are retrieved:

If path is not specified, BatchPredictionJob.score returns after the upload completes and you must download results yourself with BatchPredictionJob.download.

If path is specified, the call blocks until the job finishes. When no other jobs are processing on the targeted prediction instance, uploading, scoring, and downloading run in parallel without waiting for the full job to complete. Otherwise, the call still blocks but begins downloading scored data as soon as it is generated. Specifying path is the fastest way to get predictions.

output_settings={
    'type': 'localFile',
    'path': './predicted.csv',
}

Alternatively, omit path and download results after the job completes. If the job is not finished scoring, the call to BatchPredictionJob.download streams data scored so far and blocks until more is available.

You can poll for job completion using BatchPredictionJob.get_status or use BatchPredictionJob.wait_for_completion to wait.

import datarobot as dr

deployment_id = '5dc5b1015e6e762a6241f9aa'

job = dr.BatchPredictionJob.score(
    deployment_id,
    intake_settings={
        'type': 'localFile',
        'file': './data_to_predict.csv',
    },
    output_settings={
        'type': 'localFile',
    },
)

job.wait_for_completion()

with open('./predicted.csv', 'wb') as f:
    job.download(f)

S3 CSV output¶

S3 CSV output requires you to pass an S3 URL to the CSV file where the scored data should be saved in the url parameter. Optional parameters:

• credential_id : str, ID of stored AWS credentials (see Credential API).
• endpoint_url : str, a non-default S3 endpoint URL. Omit to use the default AWS endpoint.

output_settings={
    'type': 's3',
    'url': 's3://public-bucket/predicted.csv',
}

Most likely, the bucket is not publicly accessible for writes, but you can supply AWS credentials using these parameters:

• aws_access_key_id
• aws_secret_access_key
• aws_session_token

Save it to the Credential API. Here is an example:

# get to make sure it exists
credential_id = '5a8ac9ab07a57a0001be501f'
cred = dr.Credential.get(credential_id)

output_settings={
    'type': 's3',
    'url': 's3://private-bucket/predicted.csv',
    'credential_id': cred.credential_id,
}

JDBC output¶

Just as for the input, JDBC output requires you to create a DataStore and Credential for your database. You must also specify statement_type, which should be one of datarobot.enums.AVAILABLE_STATEMENT_TYPES:

• data_store_id : str, the ID of the external data store connected to the JDBC data source.
• table : str, the database table name.
• schema : Optional[str], the database schema name.
• catalog : Optional[str], the database catalog name (new in v2.22).
• statement_type : str, the type of insertion statement to create.
• update_columns : list[string] (optional), column names to update when statement_type is an update or upsert variant.
• where_columns : list[string] (optional), column names used in the WHERE clause when statement_type is insert or update.
• credential_id : str, ID of credentials with write access (see Credential).

# get to make sure it exists
datastore_id = '5a8ac9ab07a57a0001be5010'
data_store = dr.DataStore.get(datastore_id)

credential_id = '5a8ac9ab07a57a0001be501f'
cred = dr.Credential.get(credential_id)

output_settings = {
    'type': 'jdbc',
    'table': 'table_name',
    'schema': 'public', # optional, if supported by database
    'catalog': 'master', # optional, if supported by database
    'statement_type': 'insert',
    'data_store_id': data_store.id,
    'credential_id': cred.credential_id,
}

BigQuery output¶

Just as for the input, BigQuery requires you to create a GCS Credential to access BigQuery:

# get to make sure it exists
credential_id = '5a8ac9ab07a57a0001be501f'
cred = dr.Credential.get(credential_id)

output_settings = {
    'type': 'bigquery',
    'dataset': 'dataset_name',
    'table': 'table_name',
    'bucket': 'bucket_in_gcs',
    'credential_id': cred.credential_id,
}

Datasphere output¶

Datasphere output requires you to create a DataStore and Credential for your database. Set type to datasphere and pass the following parameters:

# get to make sure it exists
datastore_id = '5a8ac9ab07a57a0001be5010'
data_store = dr.DataStore.get(datastore_id)

credential_id = '5a8ac9ab07a57a0001be501f'
cred = dr.Credential.get(credential_id)

output_settings = {
    'type': 'datasphere',
    'table': 'table_name',
    'schema': 'DATASPHERE_SPACE_NAME',
    'data_store_id': data_store.id,
    'credential_id': cred.credential_id,
}

Azure blob storage output¶

Azure output uses the same parameters as S3 output, with an Azure blob storage URL in url.

output_settings={
    'type': 'azure',
    'url': 'https://storage_account.blob.core.windows.net/container/predicted.csv',
    'credential_id': '5a8ac9ab07a57a0001be501f',
}

Google Cloud Storage output¶

GCP output uses the same parameters as S3 output, with a GCS URL in url.

output_settings={
    'type': 'gcp',
    'url': 'gs://bucket/predicted.csv',
    'credential_id': '5a8ac9ab07a57a0001be501f',
}

Snowflake and Synapse output¶

For Snowflake and Synapse destinations, set type to snowflake or synapse and use the same parameters as JDBC output.

CSV settings¶

csv_settings is an optional dict passed to BatchPredictionJob.score that configures CSV parsing for intake and output files:

• delimiter : str (optional, default ,), field delimiter. Use the string tab for TSV. Must be a one-character string or tab.
• quotechar : str (optional, default "), character used to quote fields containing the delimiter.
• encoding : str (optional, default utf-8), file encoding (for example, shift_jis, latin_1, or mskanji).

Time series settings¶

timeseries_settings configures time series scoring for BatchPredictionJob.score and BatchPredictionJob.score_with_leaderboard_model:

• type : str, one of forecast (default), historical, or training. forecast makes predictions using forecast_point or rows without a target. historical calculates predictions for all forecast points and distances within the predictions_start_date/predictions_end_date range. training scores subsets of training data and must be used with the dss intake type.
• forecast_point : Optional[datetime.datetime], forecast point for the dataset. Inferred from the dataset if omitted. Used when type is forecast.
• predictions_start_date : Optional[datetime.datetime], start date for historical predictions. Inferred from the dataset if omitted. Used when type is historical.
• predictions_end_date : Optional[datetime.datetime], end date for historical predictions. Inferred from the dataset if omitted. Used when type is historical.
• relax_known_in_advance_features_check : bool (default False). If True, missing values in known in advance features are allowed in the forecast window.

For BatchPredictionJob.apply_time_series_data_prep_and_score, forecast_point is required when type is forecast.

Scoring parameters¶

The following optional parameters are available on BatchPredictionJob.score and related methods:

prediction_instance¶

prediction_instance overrides the prediction server connection. Defaults to the deployment or system configuration:

• hostName : str, prediction instance hostname.
• sslEnabled : bool (optional, default True). Set to False to disable SSL.
• datarobotKey : Optional[str], organization-level DataRobot key for Managed AI Cloud prediction instances.
• apiKey : Optional[str], API key for prediction requests. Defaults to the job creator's API key.

Copy a previously submitted job¶

To submit a job using parameters from a job that was previously submitted, use BatchPredictionJob.score_from_existing. The first parameter is the job ID of another job.

import datarobot as dr

previously_submitted_job_id = '5dc5b1015e6e762a6241f9aa'

dr.BatchPredictionJob.score_from_existing(
    previously_submitted_job_id,
)

Scoring an in-memory Pandas DataFrame¶

When working with DataFrames, DataRobot provides a method for scoring the data without first writing it to a CSV file and subsequently reading the data back from a CSV file: BatchPredictionJob.score_pandas <datarobot.models.BatchPredictionJob.score_pandas>.

This method also joins the computed predictions into the existing DataFrame. The first parameter is the deployment ID and the second is the DataFrame to score.

import datarobot as dr
import pandas as pd

deployment_id = '5dc5b1015e6e762a6241f9aa'

df = pd.read_csv('testdata/titanic_predict.csv')

job, df = dr.BatchPredictionJob.score_pandas(deployment_id, df)

The method returns a copy of the job status and the updated DataFrame with the predictions added. So your DataFrame will now contain the following extra columns:

• Survived_1_PREDICTION
• Survived_0_PREDICTION
• Survived_PREDICTION
• THRESHOLD
• POSITIVE_CLASS
• prediction_status

print(df)
     PassengerId  Pclass                                          Name  ... Survived_PREDICTION  THRESHOLD  POSITIVE_CLASS
0            892       3                              Kelly, Mr. James  ...                   0        0.5               1
1            893       3              Wilkes, Mrs. James (Ellen Needs)  ...                   1        0.5               1
2            894       2                     Myles, Mr. Thomas Francis  ...                   0        0.5               1
3            895       3                              Wirz, Mr. Albert  ...                   0        0.5               1
4            896       3  Hirvonen, Mrs. Alexander (Helga E Lindqvist)  ...                   1        0.5               1
..           ...     ...                                           ...  ...                 ...        ...             ...
413         1305       3                            Spector, Mr. Woolf  ...                   0        0.5               1
414         1306       1                  Oliva y Ocana, Dona. Fermina  ...                   0        0.5               1
415         1307       3                  Saether, Mr. Simon Sivertsen  ...                   0        0.5               1
416         1308       3                           Ware, Mr. Frederick  ...                   0        0.5               1
417         1309       3                      Peter, Master. Michael J  ...                   1        0.5               1

[418 rows x 16 columns]

If you don’t want all of them or if you’re not happy with the names of the added columns, they can be modified using column remapping:

import datarobot as dr
import pandas as pd

deployment_id = '5dc5b1015e6e762a6241f9aa'

df = pd.read_csv('testdata/titanic_predict.csv')

job, df = dr.BatchPredictionJob.score_pandas(
    deployment_id,
    df,
    column_names_remapping={
        'Survived_1_PREDICTION': None,       # discard column
        'Survived_0_PREDICTION': None,       # discard column
        'Survived_PREDICTION': 'predicted',  # rename column
        'THRESHOLD': None,                   # discard column
        'POSITIVE_CLASS': None,              # discard column
    },
)

Any column mapped to None will be discarded. Any column mapped to a string will be renamed. Any column not mentioned will be kept in the output untouched. Your DataFrame now contains the following extra columns:

• predicted
• prediction_status

Refer to the documentation for BatchPredictionJob.score to see the full range of available options.

Batch prediction job definitions¶

To submit a working Batch Prediction job, you must supply a variety of elements to the datarobot.models.BatchPredictionJob.score() request payload depending on what type of prediction is required. Additionally, you must consider the type of intake and output adapters used for a given job.

Every time a new batch prediction is created, the same amount of information must be stored somewhere outside of DataRobot and resubmitted every time.

NOTE¶

The name parameter must be unique across your organization. If you attempt to create multiple definitions with the same name, the request will fail. If you wish to free up a name, you must first datarobot.models.BatchPredictionJobDefinition.delete() the existing definition before creating this one. Alternatively, you can just datarobot.models.BatchPredictionJobDefinition.update() the existing definition with a new name.

For example, a request could look like:

import datarobot as dr

deployment_id = "5dc5b1015e6e762a6241f9aa"

job = dr.BatchPredictionJob.score(
    deployment_id,
    intake_settings={
        "type": "s3",
        "url": "s3://bucket/container/file.csv",
        "credential_id": "5dc5b1015e6e762a6241f9bb"
    },
    output_settings={
        "type": "s3",
        "url": "s3://bucket/container/output.csv",
        "credential_id": "5dc5b1015e6e762a6241f9bb"
    },
)

job.wait_for_completion()

with open("./predicted.csv", "wb") as f:
    job.download(f)

Job definitions¶

If your use case requires the same (or similar) type(s) of predictions to be made multiple times, you can choose to create a Job Definition of the batch prediction job and store it for future use.

The method for creating job definitions is datarobot.models.BatchPredictionJobDefinition.create(), which includes the enabled, name, and schedule parameters.

>>> import datarobot as dr
>>> job_spec = {
...    "num_concurrent": 4,
...    "deployment_id": "5dc5b1015e6e762a6241f9aa",
...    "intake_settings": {
...        "url": "s3://foobar/123",
...        "type": "s3",
...        "format": "csv",
...        "credential_id": "5dc5b1015e6e762a6241f9bb"
...    },
...    "output_settings": {
...        "url": "s3://foobar/123",
...        "type": "s3",
...        "format": "csv",
...        "credential_id": "5dc5b1015e6e762a6241f9bb"
...    },
...}
>>> definition = BatchPredictionJobDefinition.create(
...    enabled=False,
...    batch_prediction_job=job_spec,
...    name="some_definition_name",
...    schedule=None
... )
>>> definition
BatchPredictionJobDefinition(foobar)

Execute a job definition¶

Manual job execution¶

To submit a stored job definition for scoring, you can either do so on a scheduled basis, described below, or manually submit the definition ID using datarobot.models.BatchPredictionJobDefinition.run_once():

>>> import datarobot as dr
>>> definition = dr.BatchPredictionJobDefinition.get("5dc5b1015e6e762a6241f9aa")
>>> job = definition.run_once()
>>> job.wait_for_completion()

Scheduled job execution¶

A scheduled batch prediction job works just like a regular batch prediction job, but instead DataRobot handles the execution of the job.

In order to schedule the execution of a batch prediction job, a definition must first be created using datarobot.models.BatchPredictionJobDefinition.create(), or updated using datarobot.models.BatchPredictionJobDefinition.update(). In this case, enabled is set to True and a schedule payload is provided.

Alternatively, use a shorthand version with datarobot.models.BatchPredictionJobDefinition.run_on_schedule():

>>> import datarobot as dr
>>> schedule = {
...    "day_of_week": [
...        1
...    ],
...    "month": [
...        "*"
...    ],
...    "hour": [
...        16
...    ],
...    "minute": [
...        0
...    ],
...    "day_of_month": [
...        1
...    ]
...}
>>> definition = dr.BatchPredictionJob.get("5dc5b1015e6e762a6241f9aa")
>>> job = definition.run_on_schedule(schedule)

If the created job was not enabled previously, this method will also enable it.

The schedule payload¶

The schedule payload defines at what intervals the job should run, which can be combined in various ways to construct complex scheduling terms if needed. In all of the elements in the objects, you can supply either an asterisk ["*"] denoting “every” time denomination or an array of integers (e.g. [1, 2, 3]) to define a specific interval.

The schedule payload elements¶

Disable a scheduled job¶

Job definitions are only be executed by the scheduler if enabled is set to True. If you have a job definition that was previously running as a scheduled job, but should now be stopped, simply datarobot.models.BatchPredictionJobDefinition.delete() to remove it completely, or datarobot.models.BatchPredictionJobDefinition.update() it with enabled=False if you want to keep the definition, but stop the scheduled job from executing at intervals. If a job is currently running, this will finish execution regardless.

>>> import datarobot as dr
>>> definition = dr.BatchPredictionJobDefinition.get("5dc5b1015e6e762a6241f9aa")
>>> definition.delete()