Skip to content
No result to preview.

Click in-app to access the full platform documentation for your version of DataRobot.

MLOps > MLOps public preview features > MLOps reporting for unstructured models

MLOps reporting for unstructured models

Availability information

MLOps Reporting from Unstructured Models is off by default. Contact your DataRobot representative or administrator for information on enabling this feature.

Feature flag: Enable MLOps Reporting from Unstructured Models

Now available for public preview, you can report MLOps statistics from Python custom inference models created in the Custom Model Workshop with an Unstructured (Regression), Unstructured (Binary), or Unstructured (Multiclass) target type:

With this feature enabled, when you assemble an unstructured custom inference model in Python, you can read the mlops input argument from the kwargs as follows:

mlops = kwargs.get('mlops')

For an example of an unstructured Python custom model with MLOps reporting, see the DataRobot User Models repository.

Unstructured custom model reporting methods

If the value of mlops is not None, you can access and use the following methods:

report_deployment_stats

Reports the number of predictions and execution time to DataRobot MLOps.

report_deployment_stats(num_predictions: int, execution_time: float)
Argument Description
num_predictions The number of predictions.
execution_time The time, in milliseconds, that it took to calculate all predictions.

report_predictions_data

Reports the features, along with their predictions and association IDs, to DataRobot MLOps.

report_predictions_data(features_df: pandas.DataFrame, predictions: list, association_ids: list, class_names: list)
Argument Description
features_df A dataframe containing all features to track and monitor. Exclude any features you don't want to report from the dataframe.
predictions A list of predictions.
  • For regression deployments, this is a 1-dimensional list containing prediction values (e.g., [1, 2, 4, 3, 2]).
  • For classification deployments, this is a 2-dimensional list, where the inner list is the probabilities for each class type (e.g., [[0.2, 0.8], [0.3, 0.7]]).
association_ids (Optional) A list of association IDs corresponding to each prediction. Association IDs are used to calculate accuracy and must be unique for each reported prediction. The number of predictions should equal the number of association_ids in the list.
class_names (Classification only) A list of the names of predicted classes (e.g., ["class1", "class2", "class3"]). For classification deployments, the class names must be in the same order as the prediction probabilities reported. If the order isn't specified, the prediction order defaults to the order of the class names on the deployment. This argument is ignored for regression deployments.

Local testing

To test an unstructured custom model with MLOps reporting locally, you must use the drum utility with the following input arguments (or the corresponding environment variables):

Input argument Environment variable Description
--target-type TARGET_TYPE Must be unstructured.
--webserver EXTERNAL_WEB_SERVER_URL The DataRobot external web server URL.
--api-token API_TOKEN The DataRobot API token.
--monitor-embedded MLOPS_REPORTING_FROM_UNSTRUCTURED_MODELS Enables a model to use MLOps library to report statistics.
--deployment-id DEPLOYMENT_ID The deployment ID for monitoring model predictions.
--model-id MODEL_ID The deployed model ID for monitoring predictions.
Updated May 25, 2023
Back to top