LRS auto-scaling with KEDA¶

DataRobot's Long Running Service (LRS) auto-scaling feature provides dynamic horizontal scaling capabilities for custom model deployments. This feature leverages KEDA (Kubernetes Event-Driven Autoscaling) and the KEDA HTTP Add-on to automatically scale LRS instances based on HTTP workload and CPU usage, including the ability to scale to zero for cost optimization.

The auto-scaling functionality provides a standardized scaling approach that enables:

• CPU-based scaling: Scale based on CPU utilization thresholds
• HTTP-based scaling: Scale based on HTTP request metrics
• Scale-to-zero: Automatically scale down to zero replicas during periods of inactivity
• Cost optimization: Reduce infrastructure costs by only running resources when needed

Note: This feature is initially available for deployed custom models on LRS and not for other LRS-based workloads.

Prerequisites¶

LRS auto-scaling depends on KEDA and the KEDA HTTP Add-on, which are shared cluster amenities. Before enabling LRS auto-scaling, install and configure them as described in KEDA event-driven autoscaling, including the prerequisites, HTTP Add-on configuration values, network requirements, and version requirements.

Custom model (LRS) deployments use path-based HTTPScaledObject routing and have no KEDA HTTP Add-on version restriction.

LRS configuration¶

The following LRS-specific values control how the LRS controller connects to the cluster-wide KEDA and KEDA HTTP Add-on installations.

Configuration values for LRS¶

These are the default values set in the LRS chart:

Configuration values for KEDA-HTTP-add-on¶

When installing the KEDA HTTP Add-on, ensure the following default values are configured:

keda-add-ons-http:
    interceptor:
      responseHeaderTimeout: 600s
      replicas:
        waitTimeout: 300s

See additional configuration options here https://github.com/kedacore/charts/blob/main/http-add-on/Chart.yaml

Configuration values for LRS¶

These are the default values set in the LRS chart

lrs-operator:
   keda:
      enabled: false
      kedaNamespace: "keda-operator"
      kedaHTTPNamespace: "keda-http-addon"
      proxyServiceName: "keda-add-ons-http-interceptor-proxy"
      proxyServicePort: "8080"
      externalScalerName: "keda-add-ons-http-external-scaler"
      externalScalerPort: "9090"
   operator:
      ...
      config:
         scaledObjectCooldownSeconds: 86400 #  24 hours
      ...

To enable scaling, set lrs-operator.keda.enabled to true. Make sure to update the rest of the values to match your installation of KEDA and the KEDA HTTP Add-on so the LRS controller knows which namespace the components are installed in and what they're called. Set scaledObjectCooldownSeconds: 604800 which translates to scaling to zero after 7 days of idle time.

Network¶

The HTTP Add-on brings a proxy component which records metrics and routes the traffic from the ingress to the LRS. It's therefore crucial that the HTTP Add-on has access to send traffic to LRS pods.

KEDA HTTP add-on version requirements¶

DataRobot uses the KEDA HTTP Add-on for two scaling paths with different minimum versions:

Workload API protons route requests with header-based HTTPScaledObject rules (spec.headers), introduced in the KEDA HTTP Add-on v0.12.0. For scale-to-zero to work properly, clusters that run Workload API protons must install the KEDA HTTP Add-on 0.12.0 or later. Custom model (LRS) deployments use path-based routing and continue to work on earlier versions.