Skip to content
For Self-Managed AI Platform users running v11.1, see the on-premise platform documentation

NIM Containers – Troubleshooting Appendix

See the Air‑Gap configuration guide.

Profile not found

Symptom

Container exits during start‑up with NoSuchKey: 

Selected profile: 74bfd8b2df5eafe452a9887637eef4820779fb4e1edb72a4a7a2a1a2d1e6480b (tensorrt_llm-a10g-bf16-tp1-pp1-throughput)
...
Exception: S3 GetObject failed: service error: NoSuchKey: The specified key does not exist.

Root cause

The requested model profile is missing from the object‑storage bucket.

Fix

  1. Copy the Selected profile name from the log.
  2. Follow the steps to download and upload profiles:
  3. Download profiles
  4. Upload profiles

Lazy instance has previously been poisoned

Symptom

pyo3_runtime.PanicException: Lazy instance has previously been poisoned

Root cause

  • Endpoint is HTTP NIM Containers allow only communication over HTTPS.

  • Container cannot verify the HTTPS certificate Mount your Public CA or Private CA bundle to all DataRobot K8S workloads, see Custom CA.

MinIO connection error

Symptom

Exception: S3 GetObject failed: dispatch failure: io error: error trying to connect: 
dns error: failed to lookup address information: Name or service not known: dns error

Root cause

NIM supports only the S3 virtual‑hosted style (<bucket>.<domain>). MinIO must therefore accept wildcard hosts.

Fix

Option A – Wildcards (recommended) 1. DNS: *.minio.internal-example.net 2. Certificate SAN: *.minio.internal-example.net 3. Ingress: allow wildcard host. 4. MinIO server: set domain name in the environment variable 

env:
  - name: MINIO_DOMAIN
    value: minio.internal-example.net

Read also: MinIO official documentation: MINIO_DOMAIN

Option B - Path‑style fallback 1. Create a bucket that matches the domain part (minio). 2. Configure NIM: 

NIM_REPOSITORY_OVERRIDE=s3://minio/
AWS_ENDPOINT_URL=https://internal-example.net/

  1. In MinIO, set MINIO_DOMAIN=internal-example.net.

Errors during model profiles upload

Double bucket name in URL

Symptom: 

botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: 
"https://nim-bucket.nim-bucket.minio.internal-example.net/nim%252Fmeta%252Fllama-3.2-1b-instruct%253Ahf-e9f8eff-nim1.5%252B%253Ffile%253DLICENSE.txt"

Ensure AWS_ENDPOINT_URL does not include the bucket name.

# ❌ Wrong
export AWS_ENDPOINT_URL=https://nim-bucket.minio.internal-example.net/

# ✅ Correct
export AWS_ENDPOINT_URL=https://minio.internal-example.net/

4.2 Signature Does Not Match

Symptom: 

boto3.exceptions.S3UploadFailedError: Failed to upload ...
An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: 
The request signature we calculated does not match the signature you provided. 
Check your key and signing method.

Root cause: Usually caused by incorrect access key or secret, search for HTTP 403 logs