Troubleshooting¶
This page covers two layers: the Workload API control-plane services (which an administrator owns) and the workloads users run on them.
Services (control plane)¶
Start by confirming the deployed components are running, then check the logs of the relevant Deployment for the symptom:
kubectl get deploy -n <namespace>
kubectl logs deploy/workload-api-app -n <namespace>
The control-plane Deployments are workload-api-app, workload-api-task-workers, covalent-cloud-server, workload-monitor-app, and lrs-operator-manager.
| Symptom | Likely cause | Resolution |
|---|---|---|
| Workload API isn't visible to users, even though the deployment succeeded | The feature flags aren't set, or the license doesn't include the premium features. | Confirm global.workload-api.enabled: true set the flags, and check your license—see Feature flags and licensing. |
Workloads never leave scheduled/pending; status never updates |
The workload-monitor-app deployment isn't running, so pod status never flows back to the Workload API. |
Confirm workload-monitor-app is deployed (it ships with global.workload-api.enabled) and healthy; check its logs for connectivity errors to the Workload API service. |
| Workloads can't reach internal services (databases, MinIO, internal APIs) | The default egress policy blocks private (RFC 1918) ranges. | Add egress rules as described in Networking & Security. |
Autoscaling requests are rejected with 422 |
The requested policy (scale-to-zero or httpRequestsConcurrency) requires KEDA, which isn't installed. |
Install KEDA, or use a cpuAverageUtilization policy with minReplicaCount ≥ 1. See Autoscaling. |
| Workload pods fail to create with a preStop/native lifecycle error | The native preStop hook requires Kubernetes 1.32+. |
Switch to the exec preStop form. See Graceful workload shutdown. |
| Control-plane 503s or slowness under sustained load | workload-api/covalent-cloud-server are under-provisioned for the concurrency. |
Raise replica ceilings and resources. See Capacity & Resources. |
| A resource bundle doesn't appear to users | The bundle is missing "workload" in use_cases, or (GPU) its gpu_type_label has no matching node group. |
See Adding custom resource bundles and Verify bundle availability. |
| Workload API data is missing after upgrading from 11.9.x (external PCS) | Data accumulated in the MongoDB admin database. |
Run the migrateFromAdmin job. |
Workloads¶
By default, a type=service workload runs as a Long Running Service (LRS), so it depends on the LRS operator being installed and healthy—if the operator isn't running, workloads never provision. The operator renders each proton into a Kubernetes Deployment whose pods carry DataRobot identity labels. Use them to locate a workload's pods directly:
# All pods backing a workload
kubectl get pods -l datarobot-workload-id=<workload-id>
# Pods for a specific proton (one execution of that workload)
kubectl get pods -l datarobot-proton-id=<proton-id>
Stopping a workload deletes its LRS
Stopping a workload currently removes its LRS entirely—the LRS resource and all backing objects (Deployment, Service, Ingress, and any HPA / ScaledObject / HTTPScaledObject). Starting it again re-creates them from scratch. A stopped workload therefore has no Kubernetes resources; its definition persists in MongoDB and is rebuilt on the next start.
When a specific workload misbehaves, the primary diagnostic is the per-replica status endpoint:
GET /workloads/{workload_id}/protons/{proton_id}/statusDetails
It returns each replica's pod phase, container statuses, readiness conditions, and restart counts (204 if no status snapshot is available yet). The /events endpoint records DataRobot lifecycle transitions only—it does not forward Kubernetes pod events such as crash restarts or image-pull failures, so statusDetails is the source for those.
Where to find workload logs¶
A workload's stdout and stderr are captured automatically—at every lifecycle stage, including startup and crash diagnostics—and appear on the workload's Activity log > Logs tab in the Console, with no instrumentation required. This is the first place to look for application output. See View Workload logs.
Automatic logs vs. the OpenTelemetry surface
Automatic stdout/stderr capture feeds the Activity log > Logs tab only. The Workload API's OpenTelemetry collector does not scrape container stdout into the OpenTelemetry observability surface, so plain print() or unconfigured logging never reaches the traces/metrics/logs dashboards. To emit application logs as structured OpenTelemetry records, the workload must push them itself via the OTel logging handler (OTLP HTTP)—see Instrument a Workload with OpenTelemetry. For the platform-level collector architecture (the daemonset that tails pod stdout/stderr), see Observability subchart.
Infrastructure-level failures (for example, ImagePullBackOff or exec format error) do not appear in either log surface—use statusDetails for those.
How pod state maps to workload state¶
The workload-monitor service derives a workload's state from its underlying pods (evaluated in priority order). These map to the user-facing lifecycle states:
| Pod condition | Workload state |
|---|---|
A container is in CrashLoopBackOff. |
errored |
A container is in ImagePullBackOff. |
errored |
Pod phase is Failed. |
errored |
Pod phase is Succeeded. |
stopped |
Pod phase Running, all containers ready. |
running |
Pod phase Running, not all containers ready. |
launching |
Pod phase Pending, scheduled to a node. |
provisioning |
Pod phase Pending, no node yet. |
submitted |
Common workload failure modes¶
Symptom in statusDetails |
Likely cause |
|---|---|
ErrImagePull / ImagePullBackOff |
Bad imageUri, or a private registry with no matching image pull secret, or the image is blocked by image URI validation. |
exec format error crash loop |
Wrong image architecture. The platform runs linux/amd64; rebuild with docker buildx build --platform linux/amd64. |
| Container exits non-zero on startup | Application error—inspect the workload's logs (Activity log > Logs) and any OpenTelemetry logs it emits. |
Reaches running but readiness never converges |
The endpoint isn't routing; the readiness gate keeps polling until it times out. |
For detailed, user-facing workload debugging—including copy-ready commands—see the end user guides: Troubleshooting best practices, Lifecycle states, and Health and readiness.
MongoDB collections¶
The Workload API stores all workload state in MongoDB. When investigating at the data layer, these are the collections it uses:
| Collection | Contents |
|---|---|
workloads |
Workload records—the governed identity and endpoint. |
protons |
Proton records—the running execution backing a workload. |
artifacts |
Artifact definitions (image URI, port, entrypoint, environment, probes). |
artifact_repositories |
Artifact repositories. |
artifact_image_builds |
Image Build Service build metadata per artifact. |
workload_artifact_history |
Per-workload artifact and replacement history. |
replacements |
In-progress and completed workload replacements. |
workload_taskiq_results |
Async task (TaskIQ) results backend. |
For what workloads and protons represent, see the user-facing Workload concepts and Artifacts documentation.