Portable Prediction Server running modes¶

There are two model modes supported by the server: single-model (SM) and multi-model (MM). Use SM mode when only a single model package has been mounted into the Docker container inside the /opt/ml/model directory. Use MM mode in all other cases. Despite being compatible predictions-wise, SM mode provides a simplified HTTP API that does not require a model package to be identified on disk and preloads a model into memory on start.

The Docker container Filesystem directory should match the following layouts.

For SM mode:

/opt/ml/model/
└── model_5fae9a023ba73530157ebdae.mlpkg

For MM mode:

/opt/ml/model/
├── fraud
|   └── model_5fae9a023ba73530157ebdae.mlpkg
└── revenue
    ├── config.yml
    └── revenue-estimate.mlpkg

HTTP API (single-model)¶

When running in single-model mode, the Docker image exposes three HTTP endpoints:

• POST /predictions scores a given dataset.
• GET /info returns information about the loaded model.
• GET /ping ensures the tech stack is up and running.

curl -X POST http://<ip>:8080/predictions \
    -H "Content-Type: text/csv" \
    --data-binary @path/to/scoring.csv
{
  "data": [
    {
      "predictionValues": [
        {"value": 0.250833758, "label": "yes"},
        {"value": 0.749166242, "label": "no"},
      ],
      "predictionThreshold": 0.5,
      "prediction": 0.0,
      "rowId": 0
    }
  ]
}

If CSV is the preferred output, request it using the Accept: text/csv HTTP header.

curl -X POST http://<ip>:8080/predictions \
    -H "Accept: text/csv" \
    -H "Content-Type: text/csv" \
    --data-binary @path/to/scoring.csv
<target>_yes_PREDICTION,<target>_no_PREDICTION,<target>_PREDICTION,THRESHOLD,POSITIVE_CLASS
0.250833758,0.749166242,0,0.5,yes

HTTP API (multi-model)¶

In multi-model mode, the Docker image exposes the following endpoints:

• POST /deployments/:id/predictions scores a given dataset.
• GET /deployments/:id/info returns information about the loaded model.
• POST /deployments/:id uploads a model package to the container.
• DELETE /deployments/:id deletes a model package from the container.
• GET /deployments returns a list of model packages that are in the container.
• GET /ping ensures the tech stack is up and running.

The :id included in the /deployments routes above refers to the unique identifier for model packages on the disk. The ID is the directory name containing the model package. Therefore, if you have the following /opt/ml/model layout:

/opt/ml/model/
├── fraud
|   └── model_5fae9a023ba73530157ebdae.mlpkg
└── revenue
    ├── config.yml
    └── revenue-estimate.mlpkg

You may use fraud and revenue instead of :id in the /deployments set of routes.

curl -X POST http://<ip>:8080/deployments/revenue/predictions \
    -H "Content-Type: text/csv" \
    --data-binary @path/to/scoring.csv
{
  "data": [
    {
      "predictionValues": [
        {"value": 0.250833758, "label": "yes"},
        {"value": 0.749166242, "label": "no"},
      ],
      "predictionThreshold": 0.5,
      "prediction": 0.0,
      "rowId": 0
    }
  ]
}

Monitoring¶

You can monitor prediction statistics such as data drift and accuracy by creating an external deployment in the deployment inventory.

In order to connect your model package to a certain deployment, provide the deployment ID of the deployment you want to host your prediction statistics.

If you're in Single Model (SM) mode, the deployment ID has to be provided via the MLOPS_DEPLOYMENT_ID environment variable. In Multi Model (MM) mode, a special config.yml should be prepared and dropped alongside the model package with the desired deployment_id value:

deployment_id: 5fc92906ad764dde6c3264fa

If you want to track accuracy, configure it for the deployment, and then provide extra settings for the running model:

For SM mode, set the following environment variables:

• MLOPS_ASSOCIATION_ID_COLUMN=transaction_country (required)
• MLOPS_ASSOCIATION_ID_ALLOW_MISSING_VALUES=false (optional, default=false)

For MM mode, set the following properties in config.yml:

association_id_settings:
  column_name: transaction_country
  allow_missing_values: false

HTTPS support¶

By default, PPS serves predictions over an insecure listener on port 8080 (clear text HTTP over TCP). You can also serve predictions over a secure listener port 8443 (HTTP over TLS/SSL, or simply HTTPS). When the secure listener is enabled, the insecure listener becomes unavailable.

The configuration is accomplished using the environment variables described below:

• PREDICTION_API_TLS_ENABLED: The master flag that enables HTTPS listener on port 8443 and disables HTTP listener on port 8080.

  • Default: false (HTTPS disabled)

  • Valid values (case-insensitive):

    Parameter value	Interpretation
    true, yes, y, 1	true
    false, no, n, 0	false

  Note

  The flag value must be interpreted as true to enable TLS. All other PREDICTION_API_TLS_* environment variables (if passed) are ignored if this setting is not enabled.

• PREDICTION_API_TLS_CERTIFICATE: PEM-formatted content of the TLS/SSL certificate.

  • Required: Yes if PREDICTION_API_TLS_ENABLED is true, otherwise no.
  • See also: NGINX SSL certificate documentation

• PREDICTION_API_TLS_CERTIFICATE_KEY: PEM-formatted content of the secret certificate key of the TLS/SSL certificate key.

  • Required: Yes if PREDICTION_API_TLS_ENABLED is true, otherwise no.
  • See also: NGINX SSL certificate key documentation

• PREDICTION_API_TLS_CERTIFICATE_KEY_PASSWORD: Passphrase for the secret certificate key passed in PREDICTION_API_TLS_CERTIFICATE_KEY.

  • Required: Yes, only if a certificate key was created with a passphrase.

• PREDICTION_API_TLS_PROTOCOLS: Encryption protocol implementation(s) to use.

  • Default: TLSv1.2 TLSv1.3
  • Valid values: SSLv2|SSLv3|TLSv1|TLSv1.1|TLSv1.2|TLSv1.3, or any space-separated combination of these values.

  Warning

  As of August 2021, all implementations except TLSv1.2 and TLSv1.3 are considered deprecated and/or insecure. DataRobot highly recommends using only these implementations. New installations may consider using TLSv1.3 exclusively as it is the most recent and secure TLS version.

• PREDICTION_API_TLS_CIPHERS: List of cipher suites to use.

  • Default: Mandatory TLSv1.3 ciphers and recommended TLSv1.2 ciphers
  • Required: No.
  • Valid values: See OpenSSL syntax for cipher suites.

  Warning

  TLS support is an advanced feature. The cipher suites list has been carefully selected to follow the latest recommendations and current best practices. DataRobot does not recommend overriding it.

Environment variables¶

Request parameters¶

Headers¶

The PPS does not support authorization; therefore, Datarobot-key and Authorization are not needed.

Query arguments¶

The predictions routes (POST /predictions (single-model mode) and POST /deployments/:id/predictions) have the same query arguments and HTTP headers as their standard route counterparts, with a few exceptions. As with regular Dedicated Predictions API, the exact list of supported arguments depends on the deployed model. Below is the list of general query arguments supported by every deployment.

Note the following:

• You can't pass the passthroughColumns and passthroughColumnsSet parameters in the same request.
• While there is no limit on the number of column names you can pass with the passthroughColumns query parameter, there is a limit on the size of the HTTP request line (currently 8192 bytes).

Prediction Explanation parameters¶

You can parametrize the Prediction Explanations prediction request with the following query parameters:

Time series parameters¶

You can parametrize the time series prediction request using the following query parameters:

External configuration¶

You can also use the Docker image to read and set the configuration options listed in the table above (from /opt/ml/config). The file must contain <key>=<value> pairs, where each key name is a corresponding environment variable.

Examples¶

1. Run with two workers:

   docker run \
       -v /path/to/mlpkgdir:/opt/ml/model \
       -e PREDICTION_API_WORKERS=2 \
       -e PREDICTION_API_SCORING_MODEL_CACHE_MAXSIZE=32 \
       -e PREDICTION_API_PRELOAD_MODELS_ENABLED='true' \
       -e PREDICTION_API_DEPLOYED_MODEL_RESOLVER_CACHE_TTL_SEC=0 \
       datarobot/datarobot-portable-prediction-api:<version>
   

2. Run with external monitoring configured:

   docker run \
       -v /path/to/mlpkgdir:/opt/ml/model \
       -e PREDICTION_API_MONITORING_ENABLED='true' \
       -e PREDICTION_API_MONITORING_SETTINGS='<settings>' \
       datarobot/datarobot-portable-prediction-api:<version>
   

3. Run with internal monitoring configured:

   docker run \
       -v /path/to/mlpkgdir:/opt/ml/model \
       -e PREDICTION_API_MONITORING_ENABLED='true' \
       -e PREDICTION_API_MONITORING_SETTINGS='<settings>' \
       -e MONITORING_AGENT='true' \
       -e MONITORING_AGENT_DATAROBOT_APP_URL='https://app.datarobot.com/' \
       -e MONITORING_AGENT_DATAROBOT_APP_TOKEN='<token>' \
       datarobot/datarobot-portable-prediction-api:<version>
   

4. Run with HTTPS support using default protocols and ciphers:

   docker run \
       -v /path/to/mlpkgdir:/opt/ml/model \
       -p 8443:8443 \
       -e PREDICTION_API_TLS_ENABLED='true' \
       -e PREDICTION_API_TLS_CERTIFICATE="$(cat /path/to/cert.pem)" \
       -e PREDICTION_API_TLS_CERTIFICATE_KEY="$(cat /path/to/key.pem)" \
       datarobot/datarobot-portable-prediction-api:<version>
   

5. Run with Python3 interpreter only to minimize RAM footprint:

   docker run \
       -v /path/to/my_python3_model.mlpkg:/opt/ml/model \
       -e PREDICTION_API_RPC_DUAL_COMPUTE_ENABLED='false' \
       -e PYTHON3_SERVICES='true' \
       datarobot/datarobot-portable-prediction-api:<version>
   

6. Run with Python2 interpreter only to minimize RAM footprint:

   docker run \
       -v /path/to/my_python2_model.mlpkg:/opt/ml/model \
       -e PREDICTION_API_RPC_DUAL_COMPUTE_ENABLED='false' \
       datarobot/datarobot-portable-prediction-api:<version>