Use case tracking
Use case tracking¶
Use case tracking allows you to specify what you expect to accomplish by using DataRobot. You can measure success by defining the value you expect to get and tracking the actual value you receive in real time. Use cases also provide a platform for you to collect the various DataRobot assets you are using to achieve your goals and collaborate with others.
Navigate to Use Cases to view the Use Cases dashboard.
The dashboard provides details for existing use cases and allows you to navigate to any use case you currently have access to. You can search existing use cases and create new ones with the actions in the upper left.
Create a use case¶
From the dashboard, select Create a use case. This brings you to the Overview tab for a new use case. Complete the fields to provide general information about the use case.
|Name||The name of the use case.|
|Description||A basic description of what the use case is designed to accomplish.|
|Number(s) or event(s) to predict||Targets or metrics that you measure in the use case (used for information purposes only.)|
|Owner||The user who owns the use case.|
|Target dates||The dates that outline when the use case is expected to reach individual project stages. You can only set one target date for each stage.|
The Overview tab displays a stage bar, visible on every tab, to track the current progress of a use case. Update the stage at any time by clicking on one of the radio buttons.
After completing the fields, click Save in the top right corner. To undo all changes, click Cancel. Once saved, your use case is created and is accessible from the dashboard.
The Value tab offers tools to estimate the monetary value generated by using AI to achieve your use case. This data helps you prioritize use cases that are more likely to be successful and provide greater value.
Use the snap grid to estimate the business impact (Y-axis) and feasibility (X-axis) of a use case. Each axis has 5 data points: None, Low, Medium, Med-High, High. The grid also displays built-in warnings, described in the table below:
|Try to simplify||Feasibility = None||There is value in pursuing the use case, but DataRobot recommends simplifying the problem.|
|Do not attempt||Business impact = None||There is no business impact, and DataRobot does not recommend pursuing the use case.|
To change a use case's location on the grid, click on the desired section and the dot snaps to the closest datapoint. After making any changes, click Save.
Potential value estimation¶
The Potential value estimation modal assists you in estimating the expected value to receive (in actual currency) when implementing a use case. Select Raw value to manually provide an estimation, and Value calculator to calculate an estimation with the interface.
To use the value calculator, you must choose from either the Binary Classification or Regression template. Provide numeric values to answer the questions that accompany each template. The example below displays the template for a Binary Classification use case.
After completing the fields for either template, DataRobot calculates two values based on your inputs:
- Value saved annually: The expected value that can be produced by this use case each year.
- Average value per decision: The value produced by the use case each time a decision is made.
After making any calculations, click Save.
Use the raw value method to provide an estimated value manually. Specify a currency and expected annual value for the use case. Provide details for how you calculated the value in the text box.
Production value metrics¶
When a use case is moved to the In Production stage, the Value tab updates from estimating potential value to tracking realized value. The tab displays new metrics:
|Potential value||The estimated annual value for the use case.|
|Realized value||The total realized value of the use case. Realized value is calculated by multiplying "Average value per decision” by the number of predictions made in the history of deployments tied to the use case.|
|Predictions||The total number of predictions made for all deployments attached to the use case.|
|Deployment status||The aggregated health statuses of deployments attached to this use case. The icons represent service health, data drift, and accuracy. If multiple deployments are attached to a use case, the most severe status per health type is shown. If the status shows an issue, you can view the status of individual deployments on the Attachments page.|
All predictions from an attached deployment contribute to realized value, including predictions made before the deployment was attached to the use case. If a deployment is removed from a use case, realized value derived from that deployment will also be removed from the use case.
If the average value per decision is updated, the new value will apply to all realized value calculations, including value from previous predictions. If a value calculator was not used to define potential value, then realized value cannot be calculated because the use case is lacking an “Average value per decision” value. However, you can still track deployment health and predictions over time.
Two new charts appear on the value tab when you move a use case to the "In Production" stage:
A realized value chart, measuring the realized value (Y-axis) over time (X-axis).
Number of predictions, measuring the number of predictions made (Y-axis) over time (X-axis).
Modify the time range in the upper right corner of each chart.
To review or edit information displayed in the Value tab before the use case was moved to "In Production" (snap grid, value calculator, etc.), click Show potential value information below the charts.
The Assets tab provides a collection of all of the other DataRobot objects—datasets, deployments, and more—contributing to the use case.
Six types of objects can be attached to use cases:
- Model packages
- Custom models
To attach an object, click the plus sign for the type of object you want to attach. The example below outlines the workflow for adding a modeling project.
You are directed to the attachment modal, listing all objects of the chosen type that you currently have access to. In order to attach objects, click Select next to the object. You can select multiple objects at once. When you have selected the projects to attach, click Add.
When assets are added to a use case, they will be listed on the Assets tab with some metadata. Click on an individual asset name to view it in detail. Click the orange "X" to remove it.
Sharing a use case does not automatically grant access to its datasets, modeling projects, deployments, model packages, custom models, and applications. Each of these assets could have different owners as well as collaborators with different permissions.
If you need access to assets in a shared use case, hover over the asset and click Request Access.
The Owner of the asset will receive an email notification of your request. While you are awaiting approval, you will see the following message when you hover over the asset:
The Activity tab provides a way to track changes to a use case over time. Any action taken on a use case is recorded, along with who took the action and when it was taken. Navigate pages with the arrows in the top right corner.
Use case actions¶
Each use case has three actions available: edit, share, and delete.
Select the edit icon to modify the overview page for the use case.
Select the share icon to allow other users access to the use case and assign them a role.
Select the trash icon to permanently delete a use case. Once deleted, the use case cannot be recovered but associated objects will persist (datasets, deployments, etc.)
In addition to the three action icons, each use case has a Comments sections, where all users who have a use case can host discussion. The comments supporting tagging, sending email notifications to tagged users. Comments can be edited or deleted.
Select the Stage dropdown above the list to filter the list by the stage use cases are in. When filtering by stage, the contents of the chart will change based on what stage you filter by.
Filter by All stages to view basic information for each case and perform any actions:
Filter by Ideation to view the value and business impact of the use case:
Filter by In Production to view the performance of deployed use cases: