# Set up accuracy monitoring
URL: https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-settings/nxt-accuracy-settings.html

> Configure accuracy monitoring on a deployment's Accuracy Settings tab.

# Set up accuracy monitoring

You can monitor a deployment for accuracy using the Settings > Accuracy tab, which lets you analyze the performance of the model deployment over time using standard statistical measures and exportable visualizations. To configure accuracy monitoring, you must:

- Enable target monitoringon theSettings > Data drifttab.
- Select an association IDin the accuracy settings.
- Add actualsin the accuracy settings.

On a deployment's Accuracy Settings page, configure the following settings:

| Field | Description |
| --- | --- |
| Association ID |  |
| Association ID | Defines the name of the column that contains the association ID in the prediction dataset for your model. Association IDs are required for setting up accuracy monitoring in a deployment. The association ID functions as an identifier for your prediction dataset so you can later match up outcome data (also called "actuals") with those predictions. |
| Require association ID in prediction requests | Requires your prediction dataset to have a column name that matches the name you entered in the Association ID field. When enabled, you will get an error if the column is missing. This cannot be enabled with Enable automatic association ID generation for prediction rows. |
| Enable automatic association ID generation for prediction rows | With an association ID column name defined, allows DataRobot to automatically populate the association ID values. This cannot be enabled with Require association ID in prediction requests. |
| Enable automatic actuals feedback for time series models | For time series deployments that have indicated an association ID. Enables the automatic submission of actuals, so that you do not need to submit them manually via the UI or API. Once enabled, actuals can be extracted from the data used to generate predictions. As each prediction request is sent, DataRobot can extract an actual value for a given date. This is because when you send prediction rows to forecast, historical data is included. This historical data serves as the actual values for the previous prediction request. |
| Upload Actuals |  |
| Drop file(s) here or choose file | Uploads a file with actuals to monitor accuracy by matching the model's predictions with actual values. |
| Assigned features | Configures the Assigned features settings after you upload actuals. |
| Definition |  |
| Set definition | Configures the metric, measurement, and threshold definitions for accuracy monitoring. |

## Enable target monitoring

In order to enable accuracy monitoring, you must also [enable target monitoring](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-settings/nxt-data-drift-settings.html) on the Settings > Data drift page:

If target monitoring is turned off, a message displays on the Accuracy tab to remind you to enable it.

## Set an association ID

To activate the [Accuracy](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-accuracy.html) tab for a deployment, first designate an association ID in the prediction dataset. The association ID is a [foreign key](https://www.tutorialspoint.com/Foreign-Key-in-RDBMS), linking predictions with future results (or actuals). It corresponds to an event for which you want to track the outcome; For example, you may want to track a series of loans to see if any of them have defaulted or not.

> [!NOTE] Important: Association ID for monitoring agent and monitoring jobs
> You must set an association ID before making predictions to include those predictions in accuracy tracking. For [agent-monitored](https://docs.datarobot.com/en/docs/classic-ui/mlops/deployment/mlops-agent/monitoring-agent/index.html) external model deployments with challengers (and monitoring jobs for challengers), the association ID should be `__DataRobot_Internal_Association_ID__` to [report accuracy for the modelandits challengers](https://docs.datarobot.com/en/docs/classic-ui/mlops/deployment/mlops-agent/monitoring-agent/agent-use.html#report-accuracy-for-challengers).

On the Settings > Accuracy tab of a deployment, the Association ID section has a field for the column name containing the association IDs. The column name you define in the Association ID field must match the name of the column containing the association IDs in the prediction dataset for your model. Each cell for this column in your prediction dataset should contain a unique ID that pairs with a corresponding unique ID that occurs in the actuals payload.

In addition, you can enable Require association ID in prediction requests to throw an error if the column is missing from your prediction dataset when you make a prediction request. This means that current prediction requests will start producing errors if they do not include the designated association ID field. Before enabling this setting, make sure all current requests include the association ID field.

> [!NOTE] Association IDs for chat requests
> For DataRobot-deployed text generation and agentic workflow custom models that use the Bolt-on Governance API (chat completions), you can specify an association ID directly in the chat request using the `extra_body` field. Set `datarobot_association_id` in `extra_body` to use a custom association ID instead of the auto-generated one. For more information, see the [chat()hook documentation](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/structured-custom-models.html#association-id).

You can set the column name containing the association IDs on a deployment at any time, whether predictions have been made against that deployment or not. Once set, you can only update the association ID if you have not yet made predictions that include that ID. Once predictions have been made using that ID, you cannot change it. Association IDs (the contents in each row for the designated column name) must be shorter than 128 characters, or they will be truncated to that size. If truncated, uploaded actuals will require the truncated association IDs for your actuals to properly generate accuracy statistics.

## Upload actuals

You can directly upload datasets containing actuals to a deployment from the Settings > Accuracy tab (described here) or through the [API](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-settings/nxt-accuracy-settings.html#upload-actuals-with-the-api). The deployment's prediction data must correspond to the actuals you upload. Review the [row limits](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-settings/nxt-accuracy-settings.html#actuals-upload-limit) for uploading actuals before proceeding.

1. To use actuals with your deployment, in theUpload Actualssection, click+ Add data. Either upload a file directly or select a file from theData Registry. If you upload a local file, it is added to theData Registryafter successful upload. Actuals must be snapshotted in the Data Registry to use them with a deployment.
2. Once uploaded, complete the fields that appear in theUploaded Actualssection. UnderAssigned features, each field has a dropdown menu that allows you to select any of the columns from your dataset: FieldDescriptionActual ResponseDefines the column in your dataset that contains the actual values.Association IDDefines the column that contains theassociation IDs.Timestamp (optional)Defines the column that contains the date/time when the actual values were obtained, formatted according toRFC 3339(for example, 2018-04-12T23:20:50.52Z).Keep actuals without matching predictionsDetermines if DataRobot stores uploaded actuals that don't match any existing predictions by association ID. Column name matchingThe column names for the association ID in the prediction and the actuals datasets do not need to match. The only requirement is that each dataset contains a column that includes an identifier that does match the other dataset. For example, if the columnstore_idcontains the association ID in the prediction dataset that you will use to identify a row and match it to the actual result, enterstore_idin theAssociation IDsection. In theUpload Actualssection underAssigned fields, in theAssociation IDfield, enter the name of the column in the actuals dataset that contains the identifiers associated with the identifiers in the prediction dataset.
3. After you configure theAssigned fields, clickSave. When you complete this configuration process and make predictions with a dataset containing the definedAssociation ID, theAccuracypage is enabled for your deployment.

### Upload actuals with the API

This workflow outlines how to enable the Accuracy tab for deployments using the DataRobot API.

1. From theSettings > Accuracytab, locate theAssociation IDsection.
2. In theAssociation IDfield, enter the column name containing the association IDs in your prediction dataset.
3. EnableRequire association ID in prediction requests. This requires your prediction dataset to have a column name that matches the name you entered in theAssociation IDfield. You will get an error if the column is missing. NoteYou can set an association ID andnottoggle on this setting if you are sending prediction requests that do not include the association ID and you do not want them to error; however, until it is enabled, you cannot monitor accuracy for your deployment.
4. Make predictions using a dataset that includes the association ID.
5. Submit the actual valuesvia the DataRobot API. You should review therow limitsfor uploading actuals before proceeding. NoteThe actuals payload must contain theassociationIdandactualValue, with the column names for those values in the dataset defined during the upload process. If you submit multiple actuals with the same association ID value, either in the same request or a subsequent request, DataRobot updates the actuals value; however, this update doesn't recalculate the metrics previously calculated using that initial actuals value. To recalculate metrics, you canclear the deployment statisticsand reupload the actuals (or create a new deployment). Use the following snippet in the API to submit the actual values: importrequestsAPI_TOKEN=''USERNAME='johndoe@datarobot.com'DEPLOYMENT_ID='5cb314xxxxxxxxxxxa755'LOCATION='https://app.datarobot.com'defsubmit_actuals(data,deployment_id):headers={'Content-Type':'application/json','Authorization':'Token{}'.format(API_TOKEN)}url='{location}/api/v2/deployments/{deployment_id}/actuals/fromJSON/'.format(deployment_id=deployment_id,location=LOCATION)resp=requests.post(url,json=data,headers=headers)ifresp.status_code>=400:raiseRuntimeError(resp.content)returnresp.contentdefmain():deployment_id=DEPLOYMENT_IDpayload={'data':[{'actualValue':1,'associationId':'5d8138fb9600000000000000',# str},{'actualValue':0,'associationId':'5d8138fb9600000000000001',},]}submit_actuals(payload,deployment_id)print('Done')if__name__=="__main__":main() After submitting at least 100 actuals for a non-time series deployment (there is no minimum for time series deployments) and making predictions with corresponding association IDs, theAccuracytab becomes available for your deployment.

## Define accuracy monitoring notifications

For accuracy, the notification conditions relate to a [performance optimization metric](https://docs.datarobot.com/en/docs/reference/pred-ai-ref/opt-metric.html) for the underlying model in the deployment. Select from the same set of metrics that are available on the Leaderboard. You can visualize accuracy using the [Accuracy over Time graph](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-accuracy.html#accuracy-over-time-chart) and the [Predicted & Actual graph](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-accuracy.html#predicted-actual-chart). Accuracy monitoring is defined by a single accuracy rule. Every 30 seconds, the rule evaluates the deployment's accuracy. Notifications trigger when this rule is violated.

Before configuring accuracy notifications and monitoring for a deployment, set an [association ID](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-settings/nxt-accuracy-settings.html#set-an-association-id). If not set, DataRobot displays the following alert on the Settings > Accuracy tab:

> [!NOTE] Note
> Only deployment Owners can modify accuracy monitoring settings. They can set no more than one accuracy rule per deployment.Consumers cannot modify monitoring or notification settings.Users can [configure the conditions under which notifications are sent to them](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-settings/nxt-notification-settings.html).

To set up accuracy monitoring, on the Accuracy Settings page, in the Definition section, configure the settings for monitoring accuracy:

|  | Element | Description |
| --- | --- | --- |
| (1) | Metric | Defines the metric used to evaluate the accuracy of your deployment. The metrics available from the dropdown menu are the same as those supported by the Accuracy tab. |
| (2) | Measurement | Defines the unit of measurement for the accuracy metric and its thresholds. You can select value or percent from the dropdown. The value option measures the metric and thresholds by specific values, and the percent option measures by percent changed. The percent option is unavailable for model deployments that don't have training data. |
| (3) | "At Risk" / "Failing" thresholds | Sets the values or percentages that, when exceeded, trigger notifications. Two thresholds are supported: when the deployment's accuracy is "At Risk" and when it is "Failing." DataRobot provides default values for the thresholds of the first accuracy metric provided (LogLoss for classification and RMSE for regression deployments) based on the deployment's training data. Deployments without training data populate default threshold values based on their prediction data instead. If you change metrics, default values are not provided. |

> [!NOTE] Note
> Changes to thresholds affect the periods in which predictions are made across the entire history of a deployment. These updated thresholds are reflected in the performance monitoring visualizations on the [Accuracy](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-accuracy.html) tab.

---
