# Custom metrics for model selection

> Custom metrics for model selection - This AI Accelerator demonstrates how one can leverage
> DataRobot's Python client to extract predictions, compute custom metrics, and sort their DataRobot
> models accordingly.

This Markdown file sits beside the HTML page at the same path (with a `.md` suffix). It summarizes the topic and lists links for tools and LLM context.

Companion generated at `2026-05-06T18:17:09.583010+00:00` (UTC).

## Primary page

- [Custom metrics for model selection](https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-eval-metrics/ai-custom-metrics.html): Full documentation for this topic (HTML).

## Related documentation

- [Developer documentation](https://docs.datarobot.com/en/docs/api/index.html): Linked from this page.
- [Developer learning](https://docs.datarobot.com/en/docs/api/dev-learning/index.html): Linked from this page.
- [AI accelerators](https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/index.html): Linked from this page.
- [Model evaluation and metrics](https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-eval-metrics/index.html): Linked from this page.
- [out-of-the box](https://docs.datarobot.com/en/docs/reference/pred-ai-ref/opt-metric.html): Linked from this page.
- [Leaderboard](https://docs.datarobot.com/en/docs/reference/pred-ai-ref/leaderboard-ref.html): Linked from this page.
- [model insight](https://docs.datarobot.com/en/docs/classic-ui/modeling/analyze-models/index.html): Linked from this page.

## Documentation content

[Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/custom_leaderboard_metrics/custom_metrics.ipynb)

When it comes to evaluating model performance, DataRobot provides many of the standard metrics [out-of-the box](https://docs.datarobot.com/en/docs/reference/pred-ai-ref/opt-metric.html), either on the [Leaderboard](https://docs.datarobot.com/en/docs/reference/pred-ai-ref/leaderboard-ref.html) or as part of a [model insight](https://docs.datarobot.com/en/docs/classic-ui/modeling/analyze-models/index.html).

However, depending on the industry, you may need to sort your DataRobot leaderboard by a specific metric not natively supported by DataRobot. This AI Accelerator demonstrates how one can leverage DataRobot's Python client to extract predictions, compute custom metrics, and sort their DataRobot models accordingly. The topics covered are as follows:

- Setup: import libraries and connect to DataRobot
- Build models with Autopilot
- Retrieve predictions and actuals
- Sort models by Brier Skill Score (BSS)
- Sort models by Rate@Top1%
- Sort models by return-on-investment (ROI)

In addition, although sometimes difficult, assigning the ROI of utilizing machine learning can be vital for use case adoption and model implementation. Creating a payoff matrix uses a compted dollar figure rather than a machine learning metric.