Downloadable Portable Prediction Server¶

Portable Prediction Server (aka PPS) is a Docker image which represents a portable version of a Prediction Server. The feature that makes it available via DataRobot UI in DeveloperTools is called "Downloadable PPS". By default it is disabled in UI/API though, a user needs to toggle Enable Downloadable PPS Docker Image on in their settings (MLOps section) to enable it. PPS image is shipped in a release tarball along with "datarobot-runtime" image starting v5.3. Before DataRobot installation it is pushed to a private registry, which is then used for delivering images to a target Kubernetes cluster. It's expected that the registry might not be accessible by DataRobot end users, so that is why the feature was implemented. The sections below explain how it works and how to configure it.

How the feature works¶

Roughly speaking the feature makes sure that the image is uploaded from the Docker registry to the File Storage.

Namely:

• prediction-server-pps-upload cronjob periodically checks if PPS image presents in the File Storage;
• if the image isn't found then krane and imagesaver tools are used within the cronjob to upload it there;
• krane is used to pull the image from a private registry and store it to a file;
• then imagesaver tool reads the image from the file and streams its bytes to the Admin API route in Public API controller;
• Public API worker compresses the image contents and calculates image metadata on the fly, while pushing the compressed contents to the File Storage;
• at the end imagesaver tool updates image metadata in MongoDB.

When a user goes to Developer Tools page in UI, Public API either streams the compressed image data to the client or redirects to the File Storage (if the storage supports signed URLs and it is properly configured).

The whole process usually takes around 15-30 minutes. During that time a pod created for the prediction-server-pps-upload cronjob might consume up to 100% of a single CPU core as well as exactly 1 worker on one of the Public API pods stays busy. This process happens only once per installation: the same image will not be uploaded more than once after prediction-server-pps-upload cronjob is re-scheduled.

設定¶

The following Helm chart parameters controll if Downloadable PPS image feature is enabled (default) for an installation. Disabling it will lead to an error message in DataRobot UI in Developer Tools when Enable Downloadable PPS Docker Image feature flag is turned on.

prediction-server:
  pps:
    enabled: true 

In cases when the private registry used for hosting a PPS image is an ECR (Amazon Elastic Container Registry), then an additional service account annotation needs to be added to the values to enable prediction-server-pps-upload cronjob to pull images from it:

prediction-server:
  pps:
    enabled: true
    serviceAccount:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::<account_id>:role/<ecr-access-role> 

Where arn:aws:iam::<account_id>:role/<ecr-access-role> is an ARN of an IAM role that provides a pull access from the ECR.

Example IAM role for ECR pull access¶

Example role permission policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage"
            ],
            "Resource": "arn:aws:ecr:us-east-1:<account_id>:repository/*"
        }
    ]
} 

Example role trust policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<account_id>:oidc-provider/<oidc-provider-id>"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringLike": {
                    "<oidc-provider-id>:sub": "system:serviceaccount:<k8s-namespace>:pps-uploader-sa"
                }
            }
        }
    ]
} 

where <k8s-namespace> is a Kubernetes namespace DataRobot app is installed to (set to * in order to apply for all namespaces).

Please note that the target Kubernetes cluster needs to be configured to use <oidc-provider-id> OpenID Connect identity provider prior the installation. To know more about IAM roles for OIDC setup please check corresponding sections of AWS EKS and RedHat OpenShift pages.