Skip to content

サーバーレス予測

This section covers configuring serverless predictions for custom models.

Default "serverless compute" prediction environment

A default Serverless prediction environment (named "Serverless Compute") is created automatically for new organizations when CREATE_DEFAULT_SERVERLESS_PREDICTION_ENVIRONMENT setting is enabled (default: enabled) in the System Configuration > Predictions section.

To configure, modify values.yaml.

core:
  config_env_vars:
    CREATE_DEFAULT_SERVERLESS_PREDICTION_ENVIRONMENT: "True" 

Note This only impacts newly created organizations and thus not existing organizations. Those organizations must create a serverless environment themselves (Console - Prediction environments - Add prediction environment).

Deployment auto-stopping policy

Idle inference servers for deployments are auto-stopped (K8s resources are auto-scaled to zero) if INFERENCE_SERVER_AUTO_STOPPING setting is enabled (default: disabled) in the System Configuration > Predictions section. For the enabled auto-stopping policy to have an effect, the auto-stop job must also be enabled and configured.

To configure, modify values.yaml.

core:
  config_env_vars:
    INFERENCE_SERVER_AUTO_STOPPING: "True"

pred-environments-api:
  jobs:
    auto_stop:
      schedule: "*/15 * * * *"
      enabled: true
      grpc_secure: false
      dry_run: false
      inactivity: 1800 

Inference server memory for custom model deployments

The default requested memory in MB for Inference Server of a custom model deployment is controlled by INFERENCE_SERVER_MEMORY_FOR_CUSTOM_DEPLOYMENTS setting (default: 2560) in the System Configuration > Predictions section. The value is the memory request of the Inference Server compute container. Its memory limit is twice the request.

In most cases, don't set this value. The default fits the DataRobot runtime image. If you set it, don't go below 1024: the runtime needs more than 1 GiB of memory at idle, so an Inference Server with a 512 MB request (1 GiB limit) is OOMKilled and stays in CrashLoopBackOff.

To configure, modify values.yaml.

core:
  config_env_vars:
    INFERENCE_SERVER_MEMORY_FOR_CUSTOM_DEPLOYMENTS: 2560 

Remove 512 copied from earlier versions of this page

Earlier versions of this page used 512 in the example. If your values.yaml sets INFERENCE_SERVER_MEMORY_FOR_CUSTOM_DEPLOYMENTS: 512, remove the line or raise the value. Also check the value in System Configuration > Predictions: a value saved there takes precedence over values.yaml.

The setting applies only when DataRobot creates or updates an Inference Server, for example, when you deploy a model or replace it. Running Inference Servers keep their current memory until then. Deployments with their own compute memory setting use that value instead.

Number of threads for custom model deployments

The number of threads per uWSGI worker in the Inference Server of a custom model deployment is controlled by INFERENCE_SERVER_THREADS_FOR_CUSTOM_DEPLOYMENTS setting (default: 50) in the System Configuration > Predictions section. If value is 0, only process workers are used.

To configure, modify values.yaml.

core:
  config_env_vars:
    INFERENCE_SERVER_THREADS_FOR_CUSTOM_DEPLOYMENTS: 50