Secrets management service (SMS)¶

The Secrets Management Service (SMS) acts as a broker between the DataRobot platform and a client-managed secret storage backend, ensuring the following:

• Stores only secret metadata (identifiers and schema references) in DataRobot's database.
• Routes all secret values, such as credentials, API keys, and secure configurations, to and from the client's own secret storage.
• Ensures sensitive data is never stored in the DataRobot platform.

The Secrets Management Service is available starting with DataRobot 11.9. SMS supports the following secret storage backends:

• HashiCorp Vault (OSS or Enterprise)
• Azure Key Vault (DataRobot v11.11 and later)

You configure exactly one backend per deployment.

Requirements¶

Before enabling the Secrets Management Service, you must fulfill the following requirements:

• A supported secret storage backend is deployed and reachable over HTTPS from the DataRobot cluster.
• Credentials or an authentication method for the backend are provisioned for DataRobot.

For backend-specific requirements, see the sections below

HashiCorp Vault¶

If you are using HashiCorp Vault as your backend service, you must meet the following requirements:

• A HashiCorp Vault (OSS or Enterprise) instance is deployed and accessible over HTTPS from the DataRobot cluster.
• The KV v2 secrets engine is enabled.

• An authentication method is configured for DataRobot:

  • For AppRole authentication, the auth method is enabled in Vault, and a Role ID and Secret ID are provisioned.
  • For Kubernetes or JWT authentication, the auth method must be enabled in Vault before use.

• A base path is allocated in Vault for DataRobot secrets.

• (Optional) If Vault uses a private CA, the CA certificate is available in PEM format.
• (Optional) If using Vault Enterprise, the target namespace is identified. If not set, this defaults to the Vault root namespace.
• (Optional) If your auth method is configured in a different Vault Enterprise namespace than the KV secrets engine, both namespaces are identified, see Split auth and secrets namespaces.

Vault policy¶

HashiCorp Vault operates on a deny-by-default model, so you must attach a policy that grants the following permissions to the Vault role configured for SMS (or the token, if using token authentication):

If you plan to enable scoped token support, also grant:

The following HCL policy satisfies these requirements. Replace <kv-base-path> with the mount path allocated for DataRobot secrets:

path "<kv-base-path>/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "auth/token/lookup-self" {
  capabilities = ["read"]
}

path "auth/token/renew-self" {
  capabilities = ["update"]
}

Attach this policy to your auth method role (AppRole, Kubernetes, or JWT) or directly to the token before proceeding with enablement.

If you use a split auth and secrets namespace setup, attach the policy within the auth namespace instead, and prefix the KV path with the secrets namespace:

path "<secrets-namespace>/<kv-base-path>/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

Least-privilege Vault policy (optional)¶

To limit write access to only the secrets DataRobot manages, set secret_name_prefix (this step is optional for Vault) and scope the create, update, delete, and list capabilities to that prefix instead of the whole mount:

path "<kv-base-path>/data/<secret-name-prefix>/*" {
  capabilities = ["create", "read", "update", "delete"]
}

path "<kv-base-path>/metadata/<secret-name-prefix>/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

Then, separately grant read access for each path you intend to use as an external reference, because the secret name prefix for DataRobot is never applied to reference paths:

path "<kv-base-path>/data/myapp/db-creds" {
  capabilities = ["read"]
}

Add one block per referenced secret, or use a trailing /* to grant read across a whole subtree. Keep the auth/token/lookup-self and auth/token/renew-self paths from the base policy either way; they aren't scoped to <kv-base-path> and are always required.

When secret_name_prefixis not set, platform-managed secrets in DataRobot are keyed only by UUID, with no distinguishing path prefix, so this scoping isn't possible; the mount-wide policy above is the only option.

Azure Key Vault¶

If you are using Azure Key Vault as your backend service, you must meet the following requirements:

• An Azure Key Vault instance is deployed and reachable over HTTPS from the DataRobot cluster. Note its vault URI (for example, https://<vault-name>.vault.azure.net/).
• An Azure (Microsoft Entra ID) application registration or managed identity is provisioned for DataRobot, and you have its tenant ID and client (application) ID.
• The application is granted permission to manage secrets in the Key Vault (see Key Vault permissions).

• An authentication method is selected and its credentials are provisioned:

  • For client secret authentication, a client secret (password) is generated for the application registration.
  • For client certificate authentication, a certificate is registered on the application, and the PEM bundle (certificate and private key) is available.
  • For workload identity authentication, the cluster has an OIDC issuer enabled and the Azure Workload Identity webhook installed. A federated identity credential must also link from the Kubernetes service account in DataRobot to the application registration.

• A secret name prefix is chosen for DataRobot's secrets. A prefix is required for Azure Key Vault: it must start with a letter and contain only letters, digits, and hyphens, with a maximum of 80 characters.

Key Vault permissions¶

SMS creates, reads, and deletes secrets in the Key Vault. Using Azure RBAC, assign the built-in Key Vault Secrets Officer role to the application registration (or managed identity) in DataRobot, scoped to the target Key Vault. This role grants all of the secret operations SMS requires:

If your Key Vault uses access policies instead of Azure RBAC, grant the equivalent secret permissions (Get, Set, Delete, and Purge).

Enablement¶

SMS is disabled by default. To enable it, set global.secret-mgmt-service.enabled: true, then configure one of the supported storage backends, as described in the following sections:

global:
  secret-mgmt-service:
    enabled: true

Only one backend can be configured at a time. SMS selects the backend based on which configuration block you populate under storage and fails to start if more than one—or none—are configured.

When SMS is enabled, the chart automatically sets ENABLE_SECRET_MGMT_SERVICE_INTEGRATION: true on the core service, which causes DataRobot to route all credentials, API keys, and other sensitive configuration through SMS for storage in the configured backend. If you need to keep the SMS service running without activating this routing—for example, to retain access to secrets already stored in a backend—you can explicitly disable the integration:

core:
  config_env_vars:
    ENABLE_SECRET_MGMT_SERVICE_INTEGRATION: false

HashiCorp Vault¶

Configure the Vault connection under storage.vault:

secret-mgmt-service:
  storage:
    vault:
      addr: "https://<vault-address>:<vault-port>"
      auth_method: <auth-method>  # approle | k8s | jwt | token
      mount_path: "<kv-base-path>/"
      # namespace: "<vault-enterprise-namespace>"  # optional: Vault Enterprise only

If Vault uses a private CA, provide the CA certificate via a Kubernetes Secret; see Custom CA certificate.

Then add the auth-method-specific configuration described in the following sections.

AppRole authentication¶

Add the approle block under storage.vault:

secret-mgmt-service:
  storage:
    vault:
      approle:
        role_id: "<role-id>"
        mount_path: "approle/"
        token_wrapping: "false"

The AppRole Secret ID is sensitive credential material. The recommended approach is to provide it via a Kubernetes Secret, referencing it under app.secrets as the VAULT_APPROLE_SECRET_ID environment variable:

secret-mgmt-service:
  app:
    secrets:
      - name: VAULT_APPROLE_SECRET_ID
        valueFrom:
          secretKeyRef:
            name: <approle-credentials-secret>
            key: secret-id

Alternatively, you can set it directly in your values file:

secret-mgmt-service:
  storage:
    vault:
      approle:
        secret_id: "<secret-id>"

Kubernetes authentication¶

SMS presents its Kubernetes service account token; Vault verifies it by calling the Kubernetes API server directly. This authentication method is recommended for Kubernetes-native Vault deployments.

Add the k8s block under storage.vault:

secret-mgmt-service:
  storage:
    vault:
      k8s:
        mount_path: "kubernetes/"
        role: "<vault-role-name>"

JWT authentication¶

SMS presents its Kubernetes service account token; Vault verifies it independently using public keys from the Kubernetes OIDC discovery endpoint without a live call to the Kubernetes API server per request.

Add the jwt block under storage.vault:

secret-mgmt-service:
  storage:
    vault:
      jwt:
        mount_path: "jwt/"
        role: "<vault-role-name>"

Token authentication¶

Provide a static Vault token via a Kubernetes Secret, referencing it under app.secrets as the VAULT_TOKEN environment variable:

secret-mgmt-service:
  app:
    secrets:
      - name: VAULT_TOKEN
        valueFrom:
          secretKeyRef:
            name: <vault-credentials-secret>
            key: token

Alternatively, you can set it directly in your values file:

secret-mgmt-service:
  storage:
    vault:
      token: "<vault-token>"

Custom CA certificate¶

If your Vault instance uses TLS signed by a private CA, provide the PEM-encoded CA certificate via a Kubernetes Secret, referencing it under app.secrets as the VAULT_CA_CERT environment variable:

secret-mgmt-service:
  app:
    secrets:
      - name: VAULT_CA_CERT
        valueFrom:
          secretKeyRef:
            name: <vault-ca-cert-secret>
            key: ca-cert

Alternatively, you can set it directly in your values file as a single-line PEM string with \n-escaped newlines:

secret-mgmt-service:
  storage:
    vault:
      ca_cert: "-----BEGIN CERTIFICATE-----\n<base64-encoded-certificate>\n-----END CERTIFICATE-----"

Split auth and secrets namespaces¶

By default, SMS uses the same Vault Enterprise namespace (namespace) for both the auth method's login (AppRole, Kubernetes, or JWT) and the KV secrets engine. Some Vault Enterprise deployments instead configure the auth method in a different namespace than the KV secrets engine, for example, the auth method at the root namespace while secrets live in a child namespace. To support this, configure a separate namespace (auth_namespace):

secret-mgmt-service:
  storage:
    vault:
      namespace: "<secrets-namespace>"
      auth_namespace_enabled: true
      auth_namespace: "<auth-namespace>"  # empty string targets the root namespace

Remember to attach the Vault policy within the auth namespace, not the secrets namespace, when using this option.

Scoped token support¶

By default, SMS authenticates to Vault once and reuses the resulting token for all secret operations. Enabling scoped token support changes this behavior: SMS creates a short-lived child token with a limited TTL and only the policies required for each individual operation, discarding it immediately after use. This limits the blast radius if a token is ever compromised, but increases load on the Vault token API because a new token is created and revoked for every secret operation.

Enable this option only if your security requirements justify the additional Vault overhead.

To enable scoped tokens, ensure the Vault policy includes auth/token/create permissions, then add the following to your values override:

secret-mgmt-service:
  storage:
    vault:
      scoped_token_enabled: true
      scoped_token_policies:
        - "<policy-name>"  # policy to attach to each scoped token

Azure Key Vault¶

Configure the Key Vault connection under storage.azure, along with the required secret name prefix:

secret-mgmt-service:
  storage:
    secret_name_prefix: "<prefix>"  # required for Azure Key Vault
    azure:
      addr: "https://<vault-name>.vault.azure.net/"
      tenantId: "<azure-tenant-id>"
      authMethod: <auth-method>  # client_secret | client_certificate | workload_identity
      clientId: "<azure-application-client-id>"

Then add the auth-method-specific configuration described in the following sections.

Client secret authentication¶

Add the servicePrincipal block under storage.azure:

secret-mgmt-service:
  storage:
    azure:
      authMethod: client_secret
      servicePrincipal:
        clientSecret: "<client-secret>"

The client secret is sensitive credential material. The recommended approach is to provide it via a Kubernetes Secret, referencing it under app.secrets as the AKV_SP_CLIENT_SECRET environment variable:

secret-mgmt-service:
  app:
    secrets:
      - name: AKV_SP_CLIENT_SECRET
        valueFrom:
          secretKeyRef:
            name: <akv-credentials-secret>
            key: client-secret

When you supply the client secret via app.secrets, omit servicePrincipal.clientSecret from your values file.

Client certificate authentication¶

Provide a certificate registered on the application registration. The PEM bundle—the certificate and its private key—is sensitive credential material. The recommended approach is to provide it via a Kubernetes Secret, referencing it under app.secrets as the AKV_CLIENT_CERTIFICATE_PEM environment variable:

secret-mgmt-service:
  storage:
    azure:
      authMethod: client_certificate
  app:
    secrets:
      - name: AKV_CLIENT_CERTIFICATE_PEM
        valueFrom:
          secretKeyRef:
            name: <cert-secret-name>
            key: <cert-secret-key>

For example, create the secret from a local bundle file:

kubectl create secret generic <cert-secret-name> \
  --from-file=bundle.pem=<path-to-bundle.pem> \
  --namespace datarobot

In this example, set name to <cert-secret-name> and key to bundle.pem.

Workload identity authentication¶

Workload identity is recommended for production Kubernetes deployments. SMS authenticates using an Azure federated identity credential bound to its Kubernetes service account, so no client secret or certificate is stored in the cluster.

This method requires that the cluster has an OIDC issuer enabled, the Azure Workload Identity mutating admission webhook installed, and a federated identity credential that links SMS's Kubernetes service account to the application registration. The webhook is what injects the projected service account token and AZURE_* environment variables into the SMS pod; without it, the annotations the chart applies (below) have no effect.

Workload identity requires no values beyond the common ones described above (set authMethod to workload_identity and provide clientId).

When authMethod is workload_identity and the chart creates the service account (serviceAccount.create: true, the default), it automatically annotates the service account with azure.workload.identity/client-id and labels the pod with azure.workload.identity/use: "true". If you use an existing service account (serviceAccount.create: false), you must add the azure.workload.identity/client-id: <clientId> annotation to that service account yourself.

Apply the configuration¶

After updating your values override file, perform a Helm upgrade:

helm upgrade datarobot datarobot/datarobot-prime \
  -f your-values-override.yaml \
  --namespace datarobot

Verification¶

After the upgrade completes, verify that the SMS pod is running:

kubectl get pods -n datarobot -l role=secret-mgmt-service-app

Confirm the pod has passed its readiness probe:

kubectl get pod -n datarobot -l role=secret-mgmt-service-app -o jsonpath='{.items[0].status.conditions[?(@.type=="Ready")].status}'

The output should be True. If the pod is not ready, check the logs for backend connectivity or authentication errors:

kubectl logs -n datarobot -l role=secret-mgmt-service-app

Post-enablement¶

Once the service is running, DataRobot routes all secret operations through SMS:

• New secrets created in DataRobot are automatically stored in the configured backend.

• DataRobot performs a "lazy migration" of pre-existing secrets to the backend lazily on first access. No manual migration is required and consuming services observe no change in behavior. The migration follows these steps:

  1. DataRobot reads the secret from its previous internal store.
  2. The secret is written to the backend via SMS.
  3. The internal reference is updated to point to the backend.
  4. The secret is removed from legacy storage.

External vault references¶

By default, when you create a credential or secure configuration in DataRobot, its values are written to your secret storage backend through SMS and read back on demand; this is platform-managed storage mode. As an alternative, DataRobot also supports external reference storage mode, where a credential's values already exist at a path you manage in your secret storage backend, and DataRobot only stores a pointer to that path. DataRobot resolves the pointer against your secret storage backend on every read; it never copies the underlying values into its own storage.

This mode is useful when secrets are already provisioned and rotated through your existing secret storage backend workflows (for example, by another team or automation) and you want DataRobot to consume them directly rather than duplicating them into a DataRobot-managed path.

A credential's storage mode is set when it is created and cannot be changed afterward: It is either platform-managed (its values are written and stored by SMS) or an external reference (DataRobot only stores pointers). To switch a credential from one mode to another, create a new credential in the desired mode and update all references to the old one.

Deleting an external-reference credential or secure configuration only removes the pointer in DataRobot; the secret itself is left untouched in your secret storage backend. Platform-managed credentials work differently—deleting one also deletes its value from the backend because SMS wrote and owns that value.

How references work¶

An external-reference credential is made up of one or more references—the credential type requires one per field. Each reference has:

A locator is <path> or <path>#<key> (or <path>#<key.nested>), where <path> identifies where the secret lives in your backend and #<key> optionally selects a single value out of it. Exactly what <path> resolves to, and when a #<key> selector is required, depends on the backend (see the following sections).

The # in a locator is the separator between the path and the key selector. Only the first # in the string is treated as that separator; everything after it, including any further # characters, is part of the key selector. A literal # inside the path portion itself is not supported.

A . inside a #<key.nested> selector is always treated as a nesting separator, with no way to escape it. If an actual key name contains a literal . (for example, an object keyed by hostname, such as api.example.com), that key can't be selected at all; resolution fails the same way as a missing path, with nothing in the error to tell the two apart. Array indexing isn't supported either.

Every resolved value must be a scalar string; SMS rejects a resolved value that's a native JSON object or array (see "reference format is not valid" in the troubleshooting table below). Fields that expect JSON-shaped data, such as gcpKey on a Google service account credential, need that JSON stored as a single string at the referenced path, the same way you'd paste it into the field in platform-managed mode, not decomposed into a native nested object in your secret storage backend.

External references are read through the same DataRobot service identity used for all other SMS operations, so no additional policy or role is required beyond what is already granted for enablement.

HashiCorp Vault¶

For HashiCorp Vault, <path> is the path within the KV v2 mount configured for the storage backend, and the secret stored there is a JSON object of key-value pairs:

If the object at <path> has more than one key, you must supply a #<key> (or #<key.nested>) selector, otherwise, resolution fails because SMS cannot determine which key to use. For example:

myapp/db-creds#password

The default Vault policy already grants read on <kv-base-path>/*, which covers any path under that base that you reference; if you've scoped the policy down (see Least-privilege Vault policy), each referenced path needs read granted explicitly. References resolve only inside the Vault instance and KV v2 mount configured for the cluster. There is no way to reference a secret in another mount, another KV engine, or another Vault instance; expanding the policy does not help, since the mount SMS resolves against is fixed for the whole installation. To use a secret held elsewhere, copy it into the configured mount.

Azure Key Vault¶

For Azure Key Vault, <path> is the secret name in the vault, and each secret is a single opaque string rather than a JSON object:

Unlike Vault, a #<key> selector is never required for Azure Key Vault; <path> alone always resolves, since there's no ambiguity to break. Only use #<key> when the secret's value is itself JSON and you want to pull one field out of it. For example:

db-creds#password

Unlike the Vault examples above, an AKV secret name can't contain /; Azure rejects it, and SMS doesn't validate this ahead of time, so the failure surfaces slowly rather than immediately. Use a name like db-creds, not myapp/db-creds.

The Key Vault Secrets Officer role (or equivalent access policy, see Key Vault permissions) already includes read access to any secret in the vault. As with Vault, references resolve only inside the single Key Vault instance configured for the cluster; to use a secret held in another vault, copy it into the configured one first.

Register a credential with external references¶

You don't call SMS directly to register a reference. Instead, create the credential or secure configuration through the DataRobot UI or the public credentials API, the same way you would for any other credential, and choose external-reference storage mode instead of supplying values directly. DataRobot forwards the references to SMS on your behalf.

In the UI, when adding or editing a credential (Account settings > Credentials management) or a secure configuration (Admin > Secure configurations), select External reference in the storage mode toggle. For each field the credential type requires, enter a locator string instead of the value itself.

Via the public API, call POST /api/v2/credentials/ for credentials or POST /api/v2/secureConfigs/ for secure configurations, with a secretReferences array instead of the usual value fields. The two request shapes aren't identical: a credential is identified by credentialType, while a secure configuration is validated against a named schema and identifies its platform-managed values through a values array rather than top-level fields.

For a credential:

POST /api/v2/credentials/
{
  "name": "external-cred",
  "credentialType": "basic",
  "secretReferences": [
    { "fieldName": "user", "location": "myapp/db-creds#user" },
    { "fieldName": "password", "location": "myapp/db-creds#password" }
  ]
}

For a secure configuration:

POST /api/v2/secureConfigs/
{
  "name": "external-config",
  "schemaName": "Client ID and Client Secret",
  "secretReferences": [
    { "fieldName": "clientId", "location": "myapp/external-config-creds#clientId" },
    { "fieldName": "clientSecret", "location": "myapp/external-config-creds#clientSecret" }
  ]
}

A credential or secure configuration is entirely platform-managed or entirely an external reference; you cannot mix values and secretReferences, or supply a reference for only some of the fields a credential type or schema requires. This matters most for credential types with non-secret fields: an OAuth 2.0 credential, for example, requires referencing not just the client secret but also the client ID, token URL, and scope, every field the credential type requires, even the ones that aren't actually secret.

Every fieldName supplied must correspond to a field defined by the credential type or secure configuration schema, and DataRobot resolves each location against the secret storage backend at registration time to confirm it is reachable and well-formed. If any reference fails validation, the whole request is rejected and no partial credential or secure configuration is created.

To update an existing external-reference credential or secure configuration, PATCH the same endpoint with the secretReferences you want to add or replace; references are upserted by fieldName, leaving other fields untouched.

GET on the credential or secure configuration returns its storageMode (platformManaged or externalReference) and, for external references, echoes back the secretReferences array. A null value here doesn't necessarily mean the object is platform-managed: it's also returned when the caller lacks permission to see the values (a secure configuration's secretReferences is gated by the same permission as its values, so a plain viewer sees null even on a genuine external reference), and references are omitted entirely from list-view responses. Check storageMode directly rather than inferring it from whether secretReferences is null. DataRobot never returns resolved secret values through this endpoint, only the pointer, not the underlying value.

Troubleshooting reference resolution¶

SMS is an internal service; you never call it directly. When a reference fails to resolve, what you see depends on how you're registering the credential, and DataRobot deliberately limits how much detail reaches you so that error responses can't be used to discover the shape of your secret storage backend:

• In the UI, every reference-resolution failure, regardless of cause, is shown as a single, generic "Secret not found" error on the affected field. The UI does not distinguish a missing path from a permission error from a backend outage.
• Via the public API, a 422 response includes a per-field errors array, but the messages in it are still deliberately non-specific about why a given reference failed (see the following table). A 500 response carries no reference-specific detail at all; it returns only a generic failure message.

The table also covers failures at registration time (creating or editing a reference). A reference that resolved successfully at registration can still break later if the secret it points to is rotated away, deleted, or backend access is revoked; DataRobot doesn't detect this until something actually tries to read the value.

Regardless of what the UI or API surfaces, the specific underlying error (including the exact response from the secret storage backend) is always logged by SMS. If a reference fails and the reason isn't clear from the UI or API alone (which, for the cases above, it usually won't be), check the SMS pod logs:

kubectl logs -n datarobot -l role=secret-mgmt-service-app

If DataRobot's service identity itself loses the ability to authenticate to your secret storage backend (for example, an expired AppRole Secret ID or a revoked Kubernetes auth role), reference resolution fails the same way as a permission error on an individual path. Re-verify the credentials and policy described in Requirements for your backend.