Compliance¶

DataRobot automates many critical compliance tasks associated with developing a model and, by doing so, decreases time-to-deployment in highly regulated industries. You can generate, for each model, individualized documentation to provide comprehensive guidance on what constitutes effective model risk management. Then, you can download the report as an editable Microsoft Word document (.docx). The generated report includes the appropriate level of information and transparency necessitated by regulatory compliance demands.

The model compliance report is not prescriptive in format and content, but rather serves as a template in creating sufficiently rigorous model development, implementation, and usage documentation. The documentation provides evidence to show that the components of the model work as intended, the model is appropriate for its intended business purpose, and it is conceptually sound. This individualized model documentation is especially important for highly regulated industries. In the banking industry, for example, the report can help complete the Federal Reserve System's SR 11-7: Guidance on Model Risk Management. DataRobot's compliance documentation functionality accelerates this process by automating responses to the necessary documentation requirements that accompany such use cases. The Template Builder allows you to create, edit, and share custom documentation templates to fit your organization's specific needs.

Customizable documentation templates are available for non-time series, time series, and text generation (GenAI) models. Each template type has sections and components specific to the compliance needs of the model type, in addition to the standard sections and components. These compliance documentation templates can also access registered model information through key-value pairs added to the registered model version to extend the information provided in the documentation. Compliance documentation for text generation models (LLMs) exported from a playground—and added to the Registry—can report the results of any compliance tests linked to the model during the export process.

Generalized model validation workflow¶

The following is a high-level workflow of a typical model validation process. It is repeated for each new model or for a material change to an existing model (e.g., model re-fit or re-estimation). DataRobot's report satisfies Step 2 described below and, by extension, expedites the remaining steps.

1. Model owner identifies a use case and business need; model developer builds the model.

2. Owner and developer collaborate on a comprehensive “model development, implementation, and use” document that summarizes the model development process in detail.

3. The model development documentation is given to the model risk management team, with any applicable code and data.

4. Using the documentation, the model validation team replicates the process and performs a series of predefined statistical, analytical, and qualitative tests.

5. Validation team writes a comprehensive report summarizing their findings. Reportable issues require remediation. Non-reportable suggestions or recommendations demonstrate an effective challenge of the model development process.

6. Upon validation team approval, the model governance team secures stakeholder approval, tracks the remediation process, and performs ongoing model performance monitoring.

Feature considerations¶

Compliance documentation is available for the following project types:

• Binary
• Multiclass
• Regression
• Text generation
• Anomaly detection for time series projects with DataRobot models, but not for non-time series unsupervised mode.