NIM air-gap troubleshooting¶

This appendix explains common NIM errors, their causes, and fixes. For full procedures, see the Air-gap configuration guide.

Troubleshooting MinIO¶

Symptom¶

The NIM container is scheduled onto a GPU node but fails to start without emitting any logs.

Root cause¶

NIM may be stuck trying to pull a model profile from MinIO. Check which profiles NIM is trying to pull and where the request is stuck:

# Set up a MinIO alias
mc alias set myminio https://<minio-api-host> <username> <password> \
  --api S3v4 --path on --insecure

# List all the profiles in the bucket
mc ls myminio/<bucket> --insecure

# Trace access to the bucket
mc admin trace myminio --insecure

Look for 404 Not Found entries in the trace output; these indicate the requested model profile file is missing from the bucket:

2024-01-15T10:22:34.159 [404 Not Found] s3.HeadObject minio.internal-example.net/nim-models/nim%2Fnvidia%2Fnemotron-3-nano%3Ahf-52469ad%3Ffile%3Dnemo-evaluator-launcher-configs%252Flocal_nvidia_nemotron_3_nano_30b_a3b.yaml 10.0.0.1     1.481ms      ⇣  1.465888ms  ↑ 251 B ↓ 0 B

Resolution¶

If you find 404 responses for the expected profile, follow the Profile not found resolution below to re-download and re-upload the missing profile.

Profile not found¶

Symptom¶

The container terminates during startup with a NoSuchKey error:

Selected profile: 74bfd8b2df5eafe452a9887637eef4820779fb4e1edb72a4a7a2a1a2d1e6480b (tensorrt_llm-a10g-bf16-tp1-pp1-throughput)
...
Exception: S3 GetObject failed: service error: NoSuchKey: The specified key does not exist.

Root cause¶

The specified model profile isn't present in the object storage bucket.

Resolution¶

1. Copy the Selected profile name from the log output, e.g tensorrt_llm-a10g-bf16-tp1-pp1-throughput
2. Follow the documented procedures to download and upload the required profiles:
3. Download profiles
4. Upload profiles

Lazy instance previously poisoned¶

Symptom¶

pyo3_runtime.PanicException: Lazy instance has previously been poisoned

Root cause / resolution¶

• Endpoint is HTTP: NIM containers require all communication to occur over HTTPS.
• Container can't Verify HTTPS Certificate: Ensure your Public CA or Private CA bundle is mounted to all DataRobot Kubernetes workloads. Refer to the Public CA configuration example.

Errors during model profile uploads¶

Double bucket name in URL¶

Symptom¶

botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: 
"https://nim-bucket.nim-bucket.minio.internal-example.net/nim%252Fmeta%252Fllama-3.2-1b-instruct%253Ahf-e9f8eff-nim1.5%252B%253Ffile%253DLICENSE.txt"

Resolution¶

Ensure that the AWS_ENDPOINT_URL environment variable doesn't include the bucket name.

# Incorrect {: #incorrect }
export AWS_ENDPOINT_URL=https://nim-bucket.minio.internal-example.net/

# Correct {: #correct }
export AWS_ENDPOINT_URL=https://minio.internal-example.net/

Signature doesn't match¶

Symptom¶

boto3.exceptions.S3UploadFailedError: Failed to upload ...
An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: 
The request signature that was calculated does not match the signature you provided. 
Check your key and signing method.

Root cause¶

This error is typically caused by an incorrect access key or secret. Review the logs for HTTP 403 errors to identify authentication issues.