Autoscaling & operations¶
Workload autoscaling¶
Workload autoscaling behaves differently depending on whether KEDA is installed on the cluster.
| Scaling behavior | Requires KEDA? |
|---|---|
cpuAverageUtilization with minReplicaCount ≥ 1 |
No—rendered as a plain autoscaling/v2 HorizontalPodAutoscaler. |
Scale-to-zero (minReplicaCount: 0) |
Yes. |
httpRequestsConcurrency scaling metric |
Yes. |
Without KEDA, the LRS operator renders a plain HPA, and unsupported configurations (scale-to-zero or the httpRequestsConcurrency metric) are rejected with a 422. Pods deployed through the Workload API require KEDA HTTP Add-on v0.12.0 or later for header-based routing and scale-to-zero support.
To use HTTP-concurrency autoscaling, covalent-cloud-server must point at the cluster's KEDA HTTP add-on interceptor, and workload autoscaling must be enabled:
covalent-cloud-server:
workload:
enableAutoscaling: true
lrs:
kedaInterceptorFqdn: http://keda-add-ons-http-interceptor-proxy.keda-http-addon.svc.cluster.local:8080
Scale-to-zero idle cooldown
When a workload scales to zero, the LRS operator's scaledObjectCooldownSeconds and scaledObjectInitialCooldownSeconds (base-chart default 86400, i.e. 24 hours) govern how long an idle workload waits before scaling down. This value dominates idle GPU and CPU cost, and DataRobot's own environments override it (for example, 7 days in production). Tune lrs-operator.operator.config.scaledObjectCooldownSeconds for your cost and cold-start tradeoff.
Refer to the KEDA autoscaling configuration guide for installation and configuration details.
User-facing OpenTelemetry endpoint¶
The Workload API injects an OTLP endpoint into each workload pod as OTEL_EXPORTER_OTLP_ENDPOINT, allowing user workloads to emit traces, metrics, and logs to the DataRobot OTEL collector.
For instrumentation guidance, see Instrument a Workload with OpenTelemetry.
Graceful workload shutdown¶
Each workload container is configured with a graceful shutdown period (terminationGracePeriodSeconds, default 600 seconds) and a preStop hook. When a pod is terminated, the preStop hook runs first, followed by SIGTERM, giving the container time to drain connections before it is forcefully stopped. The preStop sleep must not exceed the graceful shutdown period.
The preStop sleep exists to cover an endpoint-deregistration race. When a pod begins terminating, Kubernetes removes it from the Service endpoints and sends SIGTERM at the same time, but endpoint removal propagates asynchronously—kube-proxy has to update iptables on every node, and the ingress controller has to update its upstreams. Until that propagation completes, the terminating pod can still receive new requests. The sleep keeps the container up and serving during that window so those requests aren't dropped.
The workload preStop hook and graceful-shutdown period are configured on covalent-cloud-server—which stamps them onto every LRS workload pod it schedules—through the PROTON_API_LRS_CONFIG__* environment variables:
covalent-cloud-server:
extraEnvVars:
- name: PROTON_API_LRS_CONFIG__LRS_GRACEFUL_SHUTDOWN_SECONDS
value: "600"
- name: PROTON_API_LRS_CONFIG__LRS_PRESTOP_HOOK
value: '["native", "sleep", "15"]'
The hook defaults to ["native", "sleep", "15"], which uses the Kubernetes native sleep action.
Increase the preStop sleep if terminating pods still receive requests
The default 15-second sleep is enough for most clusters, but on some it isn't—where endpoint propagation is slower (for example, slower kube-proxy/iptables convergence, or an ingress controller that lags in updating its upstreams), terminating pods keep getting routed traffic and clients see connection resets or 5xx during rollouts and scale-down. If you observe this, raise the sleep in PROTON_API_LRS_CONFIG__LRS_PRESTOP_HOOK (and LRS_GRACEFUL_SHUTDOWN_SECONDS if needed, since the sleep must not exceed it).
Native preStop requires Kubernetes 1.32+
The native sleep action requires Kubernetes 1.32 or later. On older clusters, switch to the exec form (["exec", "sleep", "15"]), which requires a sleep binary in the workload image.
Workloads should handle SIGTERM
If a workload container (or a user sidecar) ignores SIGTERM, stopping or deleting the workload waits out the full terminationGracePeriodSeconds—up to ~10 minutes in the worst case—before the pod is force-killed. Handle SIGTERM in workload code for prompt teardown.