Tutorial: Deploy an open-weight model from Hugging Face¶

Use the Workload API to deploy popular open-weight models from the Hugging Face Hub as managed, autoscaling endpoints. Describe a vLLM server in a single Workload spec and create the Workload directly. The platform provisions the GPU compute, exposes an authenticated OpenAI-compatible endpoint, and collects logs, traces, and metrics for the running service.

This path uses the vLLM library, an open source framework for LLM inference and serving. The official vllm/vllm-openai image starts an OpenAI-compatible HTTP server that downloads and loads models directly from the Hugging Face Hub. To serve a different model, change the model identifier in the artifact's entrypoint.

This guide serves openai/gpt-oss-20b on a single mid-size GPU bundle. gpt-oss-20b is an open-weight model released under the Apache 2.0 license (no access gating).

Prerequisites¶

Before deploying a Hugging Face LLM as a Workload, obtain the following:

• The DataRobot CLI (dr), authenticated against your DataRobot environment, with the Workload commands enabled:

  export DATAROBOT_CLI_FEATURE_WORKLOAD=true
  dr auth login
  

  See Manage Workloads with the CLI for the full command reference. DATAROBOT_ENDPOINT (ending in /api/v2) and DATAROBOT_API_TOKEN must also be set for the raw curl calls used later in this guide.

• A container registry the platform can pull from. The public vllm/vllm-openai image works out of the box.

• Access to a GPU compute bundle large enough for the chosen model, with capacity available in your cluster. See Choose a compute bundle.

• gpt-oss models are openly licensed (Apache 2.0) and do not require a Hugging Face token. To serve a gated model instead, a Hugging Face account and a Hugging Face access token with READ permission are required (plus model access permissions from the model author). Store that token as a DataRobot API Token credential on the Credentials Management page so it can be injected into the Workload by reference.

Choose a compute bundle¶

Start here. The artifact written in Define the Workload artifact must reference a GPU compute bundle that actually exists on the platform and is large enough for the model, so settle the hardware before writing the spec.

Discover the available compute bundles¶

A Workload's GPU model, VRAM, CPU, and memory come from the compute bundles the platform exposes. Only bundles returned by this query are valid values, and bundle IDs are specific to your platform, so read them from your own environment rather than copying an example ID. List them first:

# List the compute bundles available on the platform
curl -sS "${DATAROBOT_ENDPOINT}/mlops/compute/bundles/?useCases=workload" \
  -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}"

Keep the useCases=workload filter; without it, the response also contains bundles reserved for other DataRobot products. Each bundle reports its GPU type, VRAM, CPU, and memory. Note the bundle id (for example, gpu.medium) and set it in runtime.containerGroups[].resourceBundles, which takes exactly one bundle per container group. See Resource allocation and bundles for the full reference.

Determine the model's memory and context length¶

Two things drive GPU memory: the model weights and the KV cache, which grows with the context length (--max-model-len). The fastest way to get validated numbers is vLLM's own tooling rather than guesswork:

• vLLM recipes: recipes.vllm.ai publishes validated vllm serve commands per model and GPU, including quantization, context length, tensor-parallel size, and memory flags. Find the recipe matching the target model and the closest GPU, and reuse its flags and hardware sizing.
• Context length: The KV cache scales with --max-model-len, so reducing it (for example, to 8192) frees substantial VRAM and can let a model fit a smaller bundle. See vLLM's Conserving Memory guide for the levers: --max-model-len, --gpu-memory-utilization, --kv-cache-dtype fp8, and --tensor-parallel-size to shard across multiple GPUs.

For reference, gpt-oss-20b at --max-model-len 8192 occupies roughly 15 GB on a 24 GB GPU, leaving about 5 GB for the KV cache (on the order of 148,000 tokens).

Serve a larger model¶

gpt-oss-120b loads at 4-bit (MXFP4): its weights are roughly 60 GB, and in practice it needs 80–96 GB once the KV cache and activations are included. A single 80 GB GPU bundle (for example, an NVIDIA H100 80 GB) fits the model at a short context length such as --max-model-len 8192; a longer context pushes the KV cache toward the upper end of that range. For a long context, choose a multi-GPU bundle and add --tensor-parallel-size <N> to the vLLM entrypoint, where N matches the bundle's GPU count.

Define the Workload artifact¶

A single spec describes the whole Workload. The artifact block is the immutable definition of what runs: the image, the port, entrypoint, environment variables, and health probes (see Artifact concepts). The runtime block defines the resources and replicas it runs with. Creating the Workload from this spec creates both in one call.

The spec sets status: locked, which makes the artifact immutable and gives the Workload no expiry. Omit that line while iterating on the configuration to keep an editable draft, then lock it once settled. Locking is one-way, so review Promote to production before setting it.

# spec.yaml
name: gpt-oss-20b-vllm
importance: moderate
artifact:
  name: gpt-oss-20b-vllm-artifact
  description: Serve openai/gpt-oss-20b via vLLM's OpenAI-compatible HTTP server.
  status: locked
  spec:
    type: service
    containerGroups:
      - name: default
        containers:
          - name: vllm-server
            imageUri: vllm/vllm-openai:v0.26.0
            primary: true
            port: 8000
            entrypoint:
              - /bin/sh
              - -c
              - >
                exec vllm serve openai/gpt-oss-20b
                --host 0.0.0.0 --port 8000
                --max-model-len 8192
                --otlp-traces-endpoint ${OTEL_EXPORTER_OTLP_ENDPOINT}/v1/traces
            environmentVars:
              - name: USER
                value: vllm
              - name: HOME
                value: /tmp
              - name: HF_HOME
                value: /tmp/hf
              - name: TRANSFORMERS_CACHE
                value: /tmp/hf
              # OTEL_EXPORTER_OTLP_ENDPOINT is injected by the platform; do not set it here.
            startupProbe:
              path: /health
              port: 8000
              periodSeconds: 15
              failureThreshold: 60
            readinessProbe:
              path: /health
              port: 8000
              periodSeconds: 10
              failureThreshold: 3
            livenessProbe:
              path: /health
              port: 8000
              periodSeconds: 20
              failureThreshold: 3
              timeoutSeconds: 5
runtime:
  containerGroups:
    - name: default          # Must match artifact.spec.containerGroups[].name.
      replicaCount: 1
      resourceBundles:
        - gpu.medium           # Replace with a bundle ID from your platform; see "Choose a compute bundle."

Configure the health probes¶

A large model takes minutes to load, and the three probes serve different purposes (see Container health and readiness):

• startupProbe gates the other two probes, which is what allows a long model load without a restart. Size failureThreshold × periodSeconds to exceed the worst-case startup time: 60 × 15 s gives a 15-minute budget.
• readinessProbe controls whether the replica receives traffic. Keep it tight so a wedged replica leaves rotation without delay.
• livenessProbe restarts a container that has stopped working. vLLM runs its engine in a process separate from the HTTP server, so the engine can fail while the port still accepts connections. Readiness alone never restarts a container in that case.

Notes on the entrypoint¶

• Quantization is detected automatically. vLLM reads the quantization method from the checkpoint, so no --quantization flag is needed for gpt-oss models.
• The shell form enables variable expansion. The entrypoint runs through /bin/sh -c … exec … so ${OTEL_EXPORTER_OTLP_ENDPOINT} is expanded at runtime. An exec-form entrypoint array would pass the literal, unexpanded string. Without tracing, use the simpler exec form: entrypoint: ["vllm", "serve", "openai/gpt-oss-20b", "--host", "0.0.0.0", "--port", "8000"].
• The tracing transport is preconfigured. The platform injects the OTLP endpoint and its authentication headers; do not override them. The platform's collector accepts OTLP over HTTP only, while vLLM (and the OTel SDK) default to gRPC, so the entrypoint's --otlp-traces-endpoint flag is what selects HTTP.

environmentVars:
  - name: USER
    value: vllm
  - name: HOME
    value: /tmp
  - name: HF_HOME
    value: /tmp/hf
  - name: TRANSFORMERS_CACHE
    value: /tmp/hf
  - source: dr-credential
    name: HUGGING_FACE_HUB_TOKEN
    drCredentialId: <huggingface-credential-id>
    key: apiToken

Create and run the Workload¶

Create the Workload from the spec, then poll until it reaches running. dr workload create prints the Workload ID that the later commands need:

# Create the Workload; note the Workload ID it returns
dr workload create --spec-file spec.yaml

# Poll status until "running"
dr workload status <workload_id>

The status progresses submitted → provisioning → launching → running (see Lifecycle states). Expect the first launch to take several minutes: the image is pulled, the weights are downloaded from the Hugging Face Hub, and the model is loaded onto the GPU. Lifecycle commands (stop, start, delete, endpoint, list) are available once the Workload exists. See Manage Workloads with the CLI.

If the Workload does not start¶

Check the container logs first:

dr workload logs <workload_id> --level error --limit 100

See Best practices and troubleshooting for the fuller symptom-to-cause reference, including why errored is a terminal state rather than a transient blip to poll through.

Call the OpenAI-compatible endpoint¶

Retrieve the Workload's endpoint URL, then call it with any OpenAI-compatible client. The endpoint is served behind DataRobot authentication, so pass the DataRobot API token as the API key; requests without one are rejected with 401.

# Get the Workload endpoint URL
dr workload endpoint <workload_id>

# Call the vLLM OpenAI-compatible endpoint
from openai import OpenAI
import os

# The base URL is the workload endpoint retrieved in the previous step, with the vLLM /v1 suffix.
client = OpenAI(
    base_url="<workload-endpoint>/v1",
    api_key=os.environ["DATAROBOT_API_TOKEN"],
)

completion = client.chat.completions.create(
    model="openai/gpt-oss-20b",   # The model identifier served by vLLM.
    messages=[
        {"role": "system", "content": "You are a helpful assistant"},
        {"role": "user", "content": "Where is DataRobot headquartered?"},
    ],
    stream=False,
)

print(completion.to_json(indent=2))

Monitor the Workload¶

The platform injects the OTLP endpoint and its authentication headers into every container at start time; these are platform-managed, so do not override them.

Captured automatically, with no instrumentation required: logs (container stdout/stderr), service statistics (request count, error rate, response time, concurrency), request traces (one span per request), and resource metrics (replica count, CPU, memory, GPU utilization).

Captured because this spec sets --otlp-traces-endpoint: vLLM engine spans, one per request, carrying prompt, completion, and token counts for /v1/chat/completions plus time-to-first-token, queue time, prefill and decode times, and sampling parameters, along with a detailed startup trace useful for diagnosing slow cold starts.

Requiring additional setup: vLLM engine metrics such as KV-cache utilization and queue depth. See Collect vLLM engine metrics.

A running Workload can be viewed in Console at https://app.datarobot.com/console-nextgen/workloads/<workload_id>/overview.

Traces appear on the Monitoring > Data exploration tab. For more information, see the Data exploration documentation.

Collect vLLM engine metrics¶

This section is optional. Logs, service statistics, request traces, and resource metrics (including GPU utilization) are already collected without it. Add it only for vLLM's engine-internal metrics, such as KV-cache utilization and queue depth, on the Monitoring > OTel metrics surface.

vLLM exposes those metrics only on a Prometheus /metrics endpoint (pull-based), while the platform's OTel metrics surface ingests OTLP (push). To collect them, add a bridge as a sidecar: a second, non-primary container that scrapes vLLM's /metrics over localhost and re-exports the values over OTLP using the OpenTelemetry SDK (see Instrument a Workload with OpenTelemetry). Push the bridge image to a registry the platform can pull from, as described in Image URI validation.

# Sidecar: scrape vLLM's Prometheus /metrics and re-export as OTLP metrics via the OTel SDK.
- name: vllm-metrics-bridge
  imageUri: <your-registry>/vllm-metrics-bridge:latest
  environmentVars:
    - name: USER
      value: bridge
    - name: HOME
      value: /tmp
    # OTEL_EXPORTER_OTLP_ENDPOINT and its auth headers are injected by the platform.

Give every container in the group a resourceAllocation entry, dividing the bundle's capacity between them and keeping the totals within the bundle's CPU and RAM. memory accepts 1000-based units (B, KB, MB, GB) or a raw byte integer; Kubernetes-style binary suffixes (Mi, Gi) return a validation error. gpu sets GPU count only:

runtime:
  containerGroups:
    - name: default
      replicaCount: 1
      resourceBundles:
        - gpu.medium
      containers:
        - name: vllm-server
          resourceAllocation: {cpu: 3, memory: "22GB", gpu: 1}
        - name: vllm-metrics-bridge
          resourceAllocation: {cpu: 0.5, memory: "2GB"}

Scale the Workload¶

Update replicas or autoscaling via PATCH /workloads/{workload_id}/settings. See Runtime settings for the field reference and scaling metrics.

Two considerations apply to a model of this size. Keep minReplicaCount at 1 or higher, since HF_HOME points at the container's own ephemeral filesystem and a replica scaled from zero repeats both the image pull and the full weight download before it can serve traffic. Organization-level caps also apply to concurrent Workloads and replicas: a PATCH that would exceed either cap returns 403, so check current usage before scaling aggressively.