Create custom inference models¶
Custom inference models are user-created, pretrained models (or collection of files) that you can upload to DataRobot via the Custom Model Workshop. You can then upload a model artifact to create, test, and deploy custom inference models to DataRobot's centralized deployment hub.
Custom inference models can be assembled in one of the following ways:
Create a custom model without providing web server Scoring Code and a
start_server.shfile. This type of custom model must use a drop-in environment. Drop-in environments contain the web server Scoring Code and a
start_server.shfile used by the model. They are provided by DataRobot in the Workshop. You can also create your own drop-in custom environment.
Convert a legacy model.
Be sure to review the guidelines for preparing a custom model folder before proceeding. If any files overlap between the custom model and the environment folders, the model's files will take priority.
Once a custom model's file contents are assembled, you can test the contents locally for development purposes before uploading it to DataRobot. After you create a custom model in the workshop, you can run a testing suite from the Test tab.
Create a new custom model¶
To create a custom model, navigate to Model Registry > Custom Model Workshop and select the Models tab. This tab lists the models you have created. Click Add New Model.
This prompts a dialog box:
Name the custom model (1), indicate the name of the target feature (2), and select the target type: binary classification, regression, multiclass, anomaly detection, or unstructured (3). For binary classification models, you must provide positive and negative class labels in the designated fields.
Additionally, there are optional fields to provide the coding language used to build the model and a model description:
When you have completed all of the desired fields, click Add Custom Model to add the new custom model to the list of models available in the Workshop.
To add files to the new model, select it in the Workshop list. Navigate to the Test tab under the Model header and upload the individual files or folders that contain the model. You can browse and select multiple files at once or drag and drop. You can also pull files from an integrated repository.
Note that you must also upload web server Scoring Code and a
start_server.shfile to your model's folder unless you are pairing the model with a drop-in environment.
You can create custom inference models that support anomaly detection problems. If you choose to build one, reference the DRUM template. (Log in to GitHub before clicking this link.) When deploying custom inference anomaly detection models, note that the following functionality is not supported:
- Data drift
- Accuracy and association IDs
- Challenger models
- Humility rules
- Prediction intervals
Custom models can contain various machine learning libraries in the model code, but not every drop-in environment provided by DataRobot natively supports all libraries. However, you can manage these dependencies from the Workshop and update the base drop-in environments to support your model code.
To manage model dependencies, you must include a
requirements.txt file uploaded as part of your custom model. The text file must indicate the machine learning libraries used in the model code.
For example, consider a custom R model that uses Caret and XGBoost libraries. If this model is added to the Workshop and the R drop-in environment is selected, the base environment will only support Caret, not XGBoost. To address this, edit
requirements.txt to include the Caret and XGBoost dependencies. After re-uploading file with edits, the base environment can be installed with XGBoost, making the model available within the environment.
List the following, depending on model type, in
For R models, list the machine learning library dependencies.
For Python models, list the dependencies and any version constraints for the libraries. Supported constraint types include
>, and multiple constraints can be issued in a single entry (for example,
pandas >= 0.24, < 1.0).
Once the requirements file is updated to include dependencies and constraints, navigate to your custom model's Test tab. Upload the file under the Model > Content header. The Model Dependencies field updates to display the dependencies and constraints listed in the file.
From the Test tab, select a base drop-in environment under the Model Environment header. DataRobot warns you that a new environment must be built to account for the model dependencies. Select Build environment and DataRobot installs the required libraries and constraints to the base environment.
Once the base environment is updated, your custom model will be usable with the environment, allowing you to test, deploy, or register it.
Add new versions¶
If you want to update a model due to new package versions, different preprocessing steps, hyperparameters, and more, you can update the file contents to create a new version of the model and/or environment.
To do so, select the model from the workshop and navigate to the Test tab. Under the Model header, select Add Files. Upload the files or folders that you updated.
When you update the individual contents of a model, the minor version (1.1, 1.2, etc.) of the model automatically updates.
You can create a new major version of a model (1.0, 2.0, etc.) by selecting New Version. Choose to copy the contents of a previous version to the new version, or create an empty version and add new files to use for the model.
To upload a new version of an environment, follow this workflow.
You can now use a new version of the model or environment in addition to its previous versions. Select the iteration of the model that you want to use from the Version dropdown.
Assign training data¶
If you want to add training data to a custom inference model (which allows you to deploy it), you can do so by selecting a custom model and navigating to the Model Info tab.
The Model Info tab lists attributes about a custom inference model. Click Add Training Data.
A pop up appears, prompting you to upload training data.
Click Browse to upload training data. Optionally, you can specify the column name containing the partitioning information for your data (based on training/validation/holdout partitioning). If you plan to deploy the custom model and monitor its accuracy, specify the holdout partition in the column to establish an accuracy baseline. You can still track accuracy without specifying a partition column, however there will be no accuracy baseline. When the upload is complete, click Add Training Data.
Manage model resources¶
After creating a custom inference model, you can configure the resources the model consumes to facilitate smooth deployment and minimize potential environment errors in production.
You can monitor a custom model's resource allocation from the Assemble tab. The resource settings are listed below the deployment status.
To edit any resource settings, select the pencil icon (). Note that users can determine the maximum memory allocated for a model, but only organization admins can configure additional resource settings.
DataRobot recommends editing resource settings only if absolutely necessary. It is good practice to leave these settings as is, because increasing the memory may result in an unstable execution of the custom model. Additionally, increasing the number of replicas may not result in better performance as that is dependent on the custom model's speed.
Configure the resource allocations that appear in the modal.
|Memory||Determines the maximum amount of memory that may be allocated for a custom inference model. Note that if a model exceeds the allocated amount, it is evicted by the system. If this occurs during testing, the test is marked as a failure. If this occurs when the model is deployed, the model is automatically launched again by kubernetes.|
|Replicas||Sets the number of replicas executed in parallel to balance workloads when a custom model is running.|
Once you have fully configured the resource settings for a model, click Save. This creates a new version of the custom model with edited resource settings applied.
Convert legacy models¶
To convert legacy models:
Create a custom inference model. When DataRobot finishes creating and registering the model, the model’s Assemble tab opens.
Select a base environment (right-hand pane) where the task will run. To convert a legacy model, select the [DataRobot] Legacy Code Environment from the Base Environment dropdown.
From the Model inventory (left-hand pane), add the legacy modeling data and programs as your modeling content. You can pull files from a repository or add local files to assemble your model package.
When the file upload completes, if the model is not open source, DataRobot prompts to convert the file. Select the legacy content’s main program file and click Convert model to convert the file to a Java artifact optimized for model hosting and model serving.
Once converted, or if you encounter errors, click View Logs for a detailed log of the conversion. if you did encounter and determine the source of errors, once they are corrected and new files are made available to DataRobot, the Convert model button once again becomes active.
You can embed extra logic directly into the logs to assist in auditability for model risk or governance teams.
When conversion is successful, and before you can test the model, you must select the output dataset—the final dataset the legacy model used to make predictions. In the Model pane, use the dropdown to identify Output Dataset and Output Column (target feature). These selections will be used for output prediction verification during Custom Model Testing.
From this point, follow the instructions for custom model testing.
While you can deploy your custom inference model on an environment without testing, DataRobot strongly recommends your model pass testing before deployment.
To deploy a custom inference model, select the model you wish to deploy and navigate to the Test tab. Click the Deploy link in the middle of the screen.
After uploading your model, you are directed to the deployment information page. Most information for your custom model is automatically provided.
Under the Model header, you can provide functional validation data. This data is a partition of the model's training data, and is used to evaluate model performance.
|Inference data||Your inference data is being stored by DataRobot and is not required when uploading model code.|
|Execution endpoint||Select the prediction server to use for making predictions.|
|Association ID (optional)||The column name that contains the association ID in the prediction dataset for the model. The association ID functions as an identifier for the prediction dataset so you can later match up outcome data (also called "actuals") with those predictions. Read more about how association IDs enable accuracy tracking for a deployment here.|
|Enable data drift tracking (optional)||Compares the model's training data to its scoring data in order to analyze model performance over time. Requires training data to enable. Read more about data drift here.|
DataRobot recommends you upload training data (if available) to your deployment to enable more capabilities. To do so, click Add training data under the Learning header.
When you have added all the available data and your model is fully defined, your deployment is ready to be created. Click Create deployment at the top of the screen.
Once a custom inference model is deployed, it can make predictions using API calls to a dedicated prediction server managed by DataRobot. You can find more information about using the prediciton API in the Predictions documentation.
When you deploy a custom model, it generates log reports unique to this type of deployment, allowing you to debug custom code and troubleshoot prediction request failures from within DataRobot.
To view the logs for a deployed model, navigate to the deployment, open the actions menu, and select View Logs.
You can access two types of logs:
Runtime Logs are used to troubleshoot failed prediction requests (via the Predictions tab or the API). The logs are captured from the Docker container running the deployed custom model and contain up to 1 MB of data. The logs are cached for 5 minutes after you make a prediction request. You can re-request the logs by clicking Refresh.
Deployment logs are automatically captured if the custom model fails while deploying. The logs are stored permanently as part of the deployment.
Note that DataRobot only provides logs from inside the Docker container from which the custom model runs. Therefore, it is possible in cases where a custom model fails to deploy or fails to execute a prediction request that no logs will be available. This is because the failures occurred outside of the Docker container.
Use the Search bar to find specific references within the logs. Click Download Log to save a local copy of the logs.