DataRobot automates many critical compliance tasks associated with developing a model and, by doing so, decreases the time-to-deployment in highly regulated industries. You can generate, for each model, individualized documentation to provide comprehensive guidance on what constitutes effective model risk management. Then, you can download the report as an editable Microsoft Word document (
.docx). The generated report includes the appropriate level of information and transparency necessitated by regulatory compliance demands.
The model compliance report is not prescriptive in format and content, but rather serves as a guide in creating sufficiently rigorous model development, implementation, and use documentation. The documentation provides evidence to show that the components of the model work as intended, the model is appropriate for its intended business purpose, and it is conceptually sound. As such, the report can help with completing the Federal Reserve System's SR 11-7: Guidance on Model Risk Management.
Using the Python client, you can create custom compliance documentation templates. DataRobot’s customized templating capabilities provide flexibility to control the structure and contents of the generated documentation. Alternatively, DataRobot uses a default template to generate Compliance Documentation when no custom template is specified.
Complete the Model Compliance document¶
From the Compliance Documentation tab:
Optionally, consider unlocking the project's holdout as explained in the Documentation Content Improvement note: "Consider unlocking the project's holdout to include additional model performance detail in the compliance documentation."
Select the format of your compliance documentation. Choose:
- Automated Compliance Document to use the default template provided by DataRobot.
- Upload custom JSON file to provide a custom template for DataRobot to use in generating the documentation.
After selecting a template, click Generate Report to initiate DataRobot's report production process, which results in creation of a DOCX file. You will see an indicator that DataRobot is creating the report.
When the report is completed successfully, this indicator changes to a checkmark.
You can also generate compliance documentation from the Model Registry.
After you have generated the model compliance report, click Download and save the DOCX file to your system. Open the file and complete it as follows:
Areas of blue italic text are intended as guidance and instruction. They identify who should complete the section and provide detail of the required information.
Areas of black text are DataRobot's automatically generated model compliance text—preprocessing, performance, impact, task-specific, and DataRobot general information.
Model report updates¶
Once you generate the report, DataRobot stores it with your project for download at any time. In some cases, there are changes that would affect the report content (for example, unlocking holdout). This is fairly uncommon because report generation usually happens after model selection, which generally happens after the model has been tested against holdout. If you view the Compliance Documentation tab and there are changes that affect the report content, you are prompted to generate the report again.
Generalized model validation workflow¶
The following is a high-level workflow of a typical model validation process. It is repeated for each new model or for a material change to an existing model (e.g., model re-fit or re-estimation). DataRobot's report satisfies Step 2 described below and, by extension, expedites the remaining steps.
Model owner identifies a use case and business need; model developer builds the model.
Owner and developer collaborate on a comprehensive “model development, implementation, and use” document that summarizes the model development process in detail.
The model development documentation is given to the model risk management team, with any applicable code and data.
Using the documentation, the model validation team replicates the process and performs a series of predefined statistical, analytical, and qualitative tests.
Validation team writes a comprehensive report summarizing their findings. Reportable issues require remediation. Non-reportable suggestions or recommendations demonstrate an effective challenge of the model development process.
Upon validation team approval, the model governance team secures stakeholder approval, tracks the remediation process, and performs ongoing model performance monitoring.