Optimize custom model metrics with hyperparameter tuning¶

Access this AI accelerator on GitHub

This accelerator demonstrates how to improve DataRobot models using custom loss functions and advanced hyperparameter tuning.

In many real-world business problems, standard metrics like RMSE or Accuracy do not fully represent the true business cost. For example, in CLV prediction, overpredicting loss-making customers or underpredicting high-value customers can directly impact revenue and retention strategy. Similarly, classification models often need custom objectives such as maximizing recall at specific thresholds or minimizing false negatives.

By creating a custom metric and tuning models using that metric, you ensure the model is optimized for business value, not just statistical performance. This approach is essential when:

• Business costs are not symmetrical (overprediction and underprediction have different impacts)
• False negatives and false positives carry different risk levels
• Revenue-driven metrics matter more than standard ML scores
• Domain rules must be incorporated into optimization