Best practices and troubleshooting¶

This page collects recommendations for designing containers, hardening production Workloads, handling secrets, and recovering from common failures. The conceptual background is outlined in Artifact concepts, Workload concepts, and Lifecycle states.

Best practices¶

Consider the following when creating and running containerized workloads.

Container design¶

When working with containers, the platform polls a readiness probe to decide when a container can receive traffic. Liveness and startup probes are optional but recommended for resilient services.

Production hardening¶

Locked artifacts and explicit governance are the difference between a quick draft and a production deployment. Consider these best practices:

Security¶

Container images, secrets, and probe traffic all benefit from sensible defaults.

Troubleshooting¶

When a Workload misbehaves, use the events stream first, then drill into protons and per-replica detail.

Inspect a Workload¶

The fastest signal is the Workload object itself, then the lifecycle event log.

# Workload-level summary
curl -s "${DATAROBOT_ENDPOINT}/workloads/${WORKLOAD_ID}" \
  -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" | jq '{status, importance, artifactId, replacement}'

# Lifecycle events: what changed and when
curl -s "${DATAROBOT_ENDPOINT}/workloads/${WORKLOAD_ID}/events" \
  -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}"

Drill into protons¶

List the protons backing a Workload to see active and candidate instances and their current status.

curl -s "${DATAROBOT_ENDPOINT}/workloads/${WORKLOAD_ID}/protons" \
  -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" | jq '.data[] | {id, status, role}'

For container-level conditions, readiness, restart counts, and the startup logTail, call the per-proton status-details endpoint shown in the next section.

Per-replica readiness¶

For container-level conditions and pod phase per replica, use the dedicated status-details endpoint. This is where readiness conditions, container restart counts, and per-replica failure reasons live.

curl -s "${DATAROBOT_ENDPOINT}/workloads/${WORKLOAD_ID}/protons/${PROTON_ID}/statusDetails" \
  -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" \
  | jq '.replicas[] | {name, status, conditions, containers}'|

A 204 response means no status snapshot has arrived yet; retry shortly.

Application logs¶

Container output gives application-level context that complements lifecycle events and per-replica status. stdout and stderr are captured automatically at every lifecycle stage—startup, running, and errored—and surface on the Workload's Activity log > Logs tab. See View Workload logs for how to filter, search, and copy log output.

Workload stuck in launching¶

If the pod is scheduled but not all containers have passed readiness yet, or cluster resources are still being allocated (provisioning), check these common causes:

Check events, then per-replica statusDetails, for unmet conditions.

Workload reports errored¶

If at least one container is in CrashLoopBackOff or ImagePullBackOff, or the pod entered phase Failed, check these common causes:

errored is sticky: the Workload won't return to running until the failing pod is replaced or the failing container starts succeeding. Fix the underlying cause and allow the platform to restart the container, or trigger a replacement (see Replace and roll out).

Replacement failed or stuck¶

If a replacement reaches errored or the candidate proton stays in warming/initializing, check these common causes:

To recover from a failed replacement, fix the underlying issue in the artifact and issue another POST /workloads/{id}/replacement with the corrected artifactId. To revert to the previous artifact, issue POST /replacement with the original artifactId.

Error responses¶

The API returns structured error bodies on 4xx and 5xx responses. Validation errors (422) use the following format:

{
  "detail": [
    {
      "path": "artifact.spec.containerGroups.0.containers.0.port",
      "message": "ensure this value is greater than or equal to 1024",
      "code": "value_error"
    }
  ]
}

Common error codes: