# Tutorial: Deploy an open-weight model from Hugging Face > Tutorial: Deploy an open-weight model from Hugging Face - Serve an open-weight model as a GPU-backed > Workload using vLLM's OpenAI-compatible server. This Markdown file sits beside the HTML page at the same path (with a `.md` suffix). It summarizes the topic and lists links for tools and LLM context. Companion generated at `2026-08-05T05:57:35.116355+00:00` (UTC). ## Primary page - [Tutorial: Deploy an open-weight model from Hugging Face](https://docs.datarobot.com/en/docs/workload-api/create-workloads/deploy-llm-gpu.html.md): Full documentation for this topic (Markdown sidecar). ## Sections on this page - [Prerequisites](https://docs.datarobot.com/en/docs/workload-api/create-workloads/deploy-llm-gpu.html.md#prerequisites): In-page section heading. - [Choose a compute bundle](https://docs.datarobot.com/en/docs/workload-api/create-workloads/deploy-llm-gpu.html.md#choose-a-compute-bundle): In-page section heading. - [Discover the available compute bundles](https://docs.datarobot.com/en/docs/workload-api/create-workloads/deploy-llm-gpu.html.md#discover-the-available-compute-bundles): In-page section heading. - [Determine the model's memory and context length](https://docs.datarobot.com/en/docs/workload-api/create-workloads/deploy-llm-gpu.html.md#determine-the-models-memory-and-context-length): In-page section heading. - [Serve a larger model](https://docs.datarobot.com/en/docs/workload-api/create-workloads/deploy-llm-gpu.html.md#serve-a-larger-model): In-page section heading. - [Define the Workload artifact](https://docs.datarobot.com/en/docs/workload-api/create-workloads/deploy-llm-gpu.html.md#define-the-workload-artifact): In-page section heading. - [Configure the health probes](https://docs.datarobot.com/en/docs/workload-api/create-workloads/deploy-llm-gpu.html.md#configure-the-health-probes): In-page section heading. - [Notes on the entrypoint](https://docs.datarobot.com/en/docs/workload-api/create-workloads/deploy-llm-gpu.html.md#notes-on-the-entrypoint): In-page section heading. - [Create and run the Workload](https://docs.datarobot.com/en/docs/workload-api/create-workloads/deploy-llm-gpu.html.md#create-and-run-the-workload): In-page section heading. - [If the Workload does not start](https://docs.datarobot.com/en/docs/workload-api/create-workloads/deploy-llm-gpu.html.md#if-the-workload-does-not-start): In-page section heading. - [Call the OpenAI-compatible endpoint](https://docs.datarobot.com/en/docs/workload-api/create-workloads/deploy-llm-gpu.html.md#call-the-openai-compatible-endpoint): In-page section heading. - [Monitor the Workload](https://docs.datarobot.com/en/docs/workload-api/create-workloads/deploy-llm-gpu.html.md#monitor-the-workload): In-page section heading. - [Collect vLLM engine metrics](https://docs.datarobot.com/en/docs/workload-api/create-workloads/deploy-llm-gpu.html.md#collect-vllm-engine-metrics): In-page section heading. - [Scale the Workload](https://docs.datarobot.com/en/docs/workload-api/create-workloads/deploy-llm-gpu.html.md#scale-the-workload): In-page section heading. ## Related documentation - [Workload API](https://docs.datarobot.com/en/docs/workload-api/index.html.md): Linked from this page. - [Create Workloads](https://docs.datarobot.com/en/docs/workload-api/create-workloads/index.html.md): Linked from this page. - [Manage Workloads with the CLI](https://docs.datarobot.com/en/docs/workload-api/workload-interfaces/workload-cli/workload-cli.html.md): Linked from this page. - [Credentials Management page](https://docs.datarobot.com/en/docs/platform/acct-settings/stored-creds.html.md): Linked from this page. - [Resource allocation and bundles](https://docs.datarobot.com/en/docs/workload-api/operate-workloads/runtime-settings.html.md#resource-bundles): Linked from this page. - [Artifact concepts](https://docs.datarobot.com/en/docs/workload-api/build-artifacts/artifacts-concepts.html.md): Linked from this page. - [Promote to production](https://docs.datarobot.com/en/docs/workload-api/update-workloads/promote-production.html.md): Linked from this page. - [Container health and readiness](https://docs.datarobot.com/en/docs/workload-api/monitor-workloads/health-readiness.html.md): Linked from this page. - [Lifecycle states](https://docs.datarobot.com/en/docs/workload-api/create-workloads/workload-concepts.html.md#lifecycle-states): Linked from this page. - [Best practices and troubleshooting](https://docs.datarobot.com/en/docs/workload-api/get-started-workloads/best-practices.html.md#troubleshooting): Linked from this page. - [Data exploration](https://docs.datarobot.com/en/docs/workload-api/monitor-workloads/monitoring-ui/data-exploration.html.md): Linked from this page. - [Instrument a Workload with OpenTelemetry](https://docs.datarobot.com/en/docs/workload-api/monitor-workloads/instrument-with-otel.html.md#instrument-metrics): Linked from this page. ## Documentation content > [!NOTE] Premium feature > The Workload API is a premium feature. Contact your DataRobot representative or administrator for information on enabling the feature. Use the Workload API to deploy popular open-weight models from the [Hugging Face Hub](https://huggingface.co/models) as managed, autoscaling endpoints. Describe a vLLM server in a single Workload spec and create the Workload directly. The platform provisions the GPU compute, exposes an authenticated OpenAI-compatible endpoint, and collects logs, traces, and metrics for the running service. This path uses the [vLLM library](https://github.com/vllm-project/vllm), an open source framework for LLM inference and serving. The official `vllm/vllm-openai` image starts an [OpenAI-compatible HTTP server](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html) that downloads and loads models directly from the Hugging Face Hub. To serve a different model, change the model identifier in the artifact's entrypoint. This guide serves `openai/gpt-oss-20b` on a single mid-size GPU bundle.`gpt-oss-20b` is an open-weight model released under the Apache 2.0 license (no access gating). ## Prerequisites Before deploying a Hugging Face LLM as a Workload, obtain the following: - The DataRobot CLI (dr), authenticated against your DataRobot environment, with the Workload commands enabled: exportDATAROBOT_CLI_FEATURE_WORKLOAD=truedrauthlogin SeeManage Workloads with the CLIfor the full command reference.DATAROBOT_ENDPOINT(ending in/api/v2) andDATAROBOT_API_TOKENmust also be set for the rawcurlcalls used later in this guide. - A container registry the platform can pull from. The publicvllm/vllm-openaiimage works out of the box. - Access to a GPU compute bundle large enough for the chosen model, with capacity available in your cluster. SeeChoose a compute bundle. - gpt-ossmodels are openly licensed (Apache 2.0) and donotrequire a Hugging Face token. To serve agated modelinstead, aHugging Face accountand aHugging Face access tokenwithREADpermission are required (plusmodel access permissionsfrom the model author). Store that token as a DataRobotAPI Tokencredential on theCredentials Management pageso it can be injected into the Workload by reference. ## Choose a compute bundle Start here. The artifact written in [Define the Workload artifact](https://docs.datarobot.com/en/docs/workload-api/create-workloads/deploy-llm-gpu.html.md#define-the-workload-artifact) must reference a GPU compute bundle that actually exists on the platform and is large enough for the model, so settle the hardware before writing the spec. ### Discover the available compute bundles A Workload's GPU model, VRAM, CPU, and memory come from the compute bundles the platform exposes. Only bundles returned by this query are valid values, and bundle IDs are specific to your platform, so read them from your own environment rather than copying an example ID. List them first: ``` # List the compute bundles available on the platform curl -sS "${DATAROBOT_ENDPOINT}/mlops/compute/bundles/?useCases=workload" \ -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" ``` Keep the `useCases=workload` filter; without it, the response also contains bundles reserved for other DataRobot products. Each bundle reports its GPU type, VRAM, CPU, and memory. Note the bundle `id` (for example, `gpu.medium`) and set it in `runtime.containerGroups[].resourceBundles`, which takes exactly one bundle per container group. See [Resource allocation and bundles](https://docs.datarobot.com/en/docs/workload-api/operate-workloads/runtime-settings.html.md#resource-bundles) for the full reference. ### Determine the model's memory and context length Two things drive GPU memory: the model weights and the KV cache, which grows with the context length ( `--max-model-len`). The fastest way to get validated numbers is vLLM's own tooling rather than guesswork: - vLLM recipes : recipes.vllm.ai publishes validated vllm serve commands per model and GPU, including quantization, context length, tensor-parallel size, and memory flags. Find the recipe matching the target model and the closest GPU, and reuse its flags and hardware sizing. - Context length : The KV cache scales with --max-model-len , so reducing it (for example, to 8192 ) frees substantial VRAM and can let a model fit a smaller bundle. See vLLM's Conserving Memory guide for the levers: --max-model-len , --gpu-memory-utilization , --kv-cache-dtype fp8 , and --tensor-parallel-size to shard across multiple GPUs. For reference, `gpt-oss-20b` at `--max-model-len 8192` occupies roughly 15 GB on a 24 GB GPU, leaving about 5 GB for the KV cache (on the order of 148,000 tokens). > [!NOTE] MXFP4 quantization requires a recent GPU architecture > `gpt-oss` models are shipped MXFP4-quantized. MXFP4 requires CUDA compute capability 8.0 or higher: T4 bundles cannot serve these models regardless of available VRAM, while Ampere and Ada GPUs (A10G, L40S, A100) serve them through vLLM's Marlin kernels. > [!TIP] Estimate without running anything > vLLM recipes are the most reliable source for serving footprint because they account for the KV cache and are validated on real GPUs. For a quick pre-check, the following actively maintained tools help: > > Hugging Face Model Memory Utility > (or > accelerate estimate-memory > on the CLI): authoritative for > weights and activations > , but does not model the KV cache. > NyxKrage LLM Model VRAM Calculator > : enter model, quantization, > context length > , and GPU. It includes the KV cache, so it answers whether the model fits at a given context length. > Vokturz "Can You Run It?" > : a quick model-versus-GPU fit check. > > Whatever the estimate, add 10–20% headroom for overhead, and confirm on the target bundle before locking the artifact. ### Serve a larger model `gpt-oss-120b` loads at 4-bit (MXFP4): its weights are roughly 60 GB, and in practice it needs 80–96 GB once the KV cache and activations are included. A single 80 GB GPU bundle (for example, an NVIDIA H100 80 GB) fits the model at a short context length such as `--max-model-len 8192`; a longer context pushes the KV cache toward the upper end of that range. For a long context, choose a multi-GPU bundle and add `--tensor-parallel-size ` to the vLLM entrypoint, where `N` matches the bundle's GPU count. ## Define the Workload artifact A single spec describes the whole Workload. The `artifact` block is the immutable definition of what runs: the image, the port, entrypoint, environment variables, and health probes (see [Artifact concepts](https://docs.datarobot.com/en/docs/workload-api/build-artifacts/artifacts-concepts.html.md)). The `runtime` block defines the resources and replicas it runs with. Creating the Workload from this spec creates both in one call. The spec sets `status: locked`, which makes the artifact immutable and gives the Workload no expiry. Omit that line while iterating on the configuration to keep an editable `draft`, then lock it once settled. Locking is one-way, so review [Promote to production](https://docs.datarobot.com/en/docs/workload-api/update-workloads/promote-production.html.md) before setting it. ``` # spec.yaml name: gpt-oss-20b-vllm importance: moderate artifact: name: gpt-oss-20b-vllm-artifact description: Serve openai/gpt-oss-20b via vLLM's OpenAI-compatible HTTP server. status: locked spec: type: service containerGroups: - name: default containers: - name: vllm-server imageUri: vllm/vllm-openai:v0.26.0 primary: true port: 8000 entrypoint: - /bin/sh - -c - > exec vllm serve openai/gpt-oss-20b --host 0.0.0.0 --port 8000 --max-model-len 8192 --otlp-traces-endpoint ${OTEL_EXPORTER_OTLP_ENDPOINT}/v1/traces environmentVars: - name: USER value: vllm - name: HOME value: /tmp - name: HF_HOME value: /tmp/hf - name: TRANSFORMERS_CACHE value: /tmp/hf # OTEL_EXPORTER_OTLP_ENDPOINT is injected by the platform; do not set it here. startupProbe: path: /health port: 8000 periodSeconds: 15 failureThreshold: 60 readinessProbe: path: /health port: 8000 periodSeconds: 10 failureThreshold: 3 livenessProbe: path: /health port: 8000 periodSeconds: 20 failureThreshold: 3 timeoutSeconds: 5 runtime: containerGroups: - name: default # Must match artifact.spec.containerGroups[].name. replicaCount: 1 resourceBundles: - gpu.medium # Replace with a bundle ID from your platform; see "Choose a compute bundle." ``` > [!NOTE] Container requirements > The primary container's `port` must be `>= 1024`, and the container must actually listen on it (the `--port` flag in the vLLM entrypoint); non-primary containers must omit `port`. The image must include a `linux/amd64` manifest, since an ARM64-only image crash-loops with `exec format error`. DataRobot runs Workload containers as a non-root user, and the `vllm/vllm-openai` image has no password-database entry for that user, so the `USER` and `HOME` environment variables set in the preceding spec are required, not optional: omitting them produces a `getpwuid()` error at startup. See [Container requirements](https://docs.datarobot.com/en/docs/workload-api/build-artifacts/artifacts-concepts.html.md#container-requirements) and [Environment variable types](https://docs.datarobot.com/en/docs/workload-api/build-artifacts/artifacts-concepts.html.md#environment-variable-types) for the full reference. ### Configure the health probes A large model takes minutes to load, and the three probes serve different purposes (see [Container health and readiness](https://docs.datarobot.com/en/docs/workload-api/monitor-workloads/health-readiness.html.md)): - startupProbe gates the other two probes, which is what allows a long model load without a restart. Size failureThreshold × periodSeconds to exceed the worst-case startup time: 60 × 15 s gives a 15-minute budget. - readinessProbe controls whether the replica receives traffic. Keep it tight so a wedged replica leaves rotation without delay. - livenessProbe restarts a container that has stopped working. vLLM runs its engine in a process separate from the HTTP server, so the engine can fail while the port still accepts connections. Readiness alone never restarts a container in that case. > [!WARNING] Never set a liveness probe without a startup probe > Without a startup probe, a liveness probe's short delay kills the container mid-load and produces an endless crash loop. ### Notes on the entrypoint - Quantization is detected automatically. vLLM reads the quantization method from the checkpoint, so no --quantization flag is needed for gpt-oss models. - The shell form enables variable expansion. The entrypoint runs through /bin/sh -c … exec … so ${OTEL_EXPORTER_OTLP_ENDPOINT} is expanded at runtime. An exec-form entrypoint array would pass the literal, unexpanded string. Without tracing, use the simpler exec form: entrypoint: ["vllm", "serve", "openai/gpt-oss-20b", "--host", "0.0.0.0", "--port", "8000"] . - The tracing transport is preconfigured. The platform injects the OTLP endpoint and its authentication headers; do not override them. The platform's collector accepts OTLP over HTTP only, while vLLM (and the OTel SDK) default to gRPC, so the entrypoint's --otlp-traces-endpoint flag is what selects HTTP. > [!NOTE] Serving a different model > Change the model identifier in the `entrypoint` and adjust the GPU bundle to match its memory footprint, keeping `--tensor-parallel-size` equal to the bundle's GPU count (omit it for a single GPU). For a gated model, never hardcode the Hugging Face token. Add it as a credential-backed environment variable: > > ``` > environmentVars: > - name: USER > value: vllm > - name: HOME > value: /tmp > - name: HF_HOME > value: /tmp/hf > - name: TRANSFORMERS_CACHE > value: /tmp/hf > - source: dr-credential > name: HUGGING_FACE_HUB_TOKEN > drCredentialId: > key: apiToken > ``` > > Any [vLLMserveflag](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html), such as `--max-model-len` or `--gpu-memory-utilization`, can be passed as an additional entrypoint argument. ## Create and run the Workload Create the Workload from the spec, then poll until it reaches `running`.`dr workload create` prints the Workload ID that the later commands need: ``` # Create the Workload; note the Workload ID it returns dr workload create --spec-file spec.yaml # Poll status until "running" dr workload status ``` The status progresses `submitted` → `provisioning` → `launching` → `running` (see [Lifecycle states](https://docs.datarobot.com/en/docs/workload-api/create-workloads/workload-concepts.html.md#lifecycle-states)). Expect the first launch to take several minutes: the image is pulled, the weights are downloaded from the Hugging Face Hub, and the model is loaded onto the GPU. Lifecycle commands ( `stop`, `start`, `delete`, `endpoint`, `list`) are available once the Workload exists. See [Manage Workloads with the CLI](https://docs.datarobot.com/en/docs/workload-api/workload-interfaces/workload-cli/workload-cli.html.md). ### If the Workload does not start Check the container logs first: ``` dr workload logs --level error --limit 100 ``` See [Best practices and troubleshooting](https://docs.datarobot.com/en/docs/workload-api/get-started-workloads/best-practices.html.md#troubleshooting) for the fuller symptom-to-cause reference, including why `errored` is a terminal state rather than a transient blip to poll through. ## Call the OpenAI-compatible endpoint Retrieve the Workload's endpoint URL, then call it with any OpenAI-compatible client. The endpoint is served behind DataRobot authentication, so pass the DataRobot API token as the API key; requests without one are rejected with `401`. ``` # Get the Workload endpoint URL dr workload endpoint ``` ``` # Call the vLLM OpenAI-compatible endpoint from openai import OpenAI import os # The base URL is the workload endpoint retrieved in the previous step, with the vLLM /v1 suffix. client = OpenAI( base_url="/v1", api_key=os.environ["DATAROBOT_API_TOKEN"], ) completion = client.chat.completions.create( model="openai/gpt-oss-20b", # The model identifier served by vLLM. messages=[ {"role": "system", "content": "You are a helpful assistant"}, {"role": "user", "content": "Where is DataRobot headquartered?"}, ], stream=False, ) print(completion.to_json(indent=2)) ``` > [!NOTE] Model name > With the vLLM OpenAI-compatible server, the `model` value in the request is the model identifier the server was started with (the value in the artifact `entrypoint`); for example, `openai/gpt-oss-20b`. Alongside `/v1/chat/completions`, the vLLM image also serves `/v1/completions`, `/v1/responses`, `/v1/messages`, `/v1/models`, and its own Prometheus `/metrics` endpoint. ## Monitor the Workload The platform injects the OTLP endpoint and its authentication headers into every container at start time; these are platform-managed, so do not override them. Captured automatically, with no instrumentation required: logs (container stdout/stderr), service statistics (request count, error rate, response time, concurrency), request traces (one span per request), and resource metrics (replica count, CPU, memory, GPU utilization). Captured because this spec sets `--otlp-traces-endpoint`: vLLM engine spans, one per request, carrying prompt, completion, and token counts for `/v1/chat/completions` plus time-to-first-token, queue time, prefill and decode times, and sampling parameters, along with a detailed startup trace useful for diagnosing slow cold starts. Requiring additional setup: vLLM engine metrics such as KV-cache utilization and queue depth. See [Collect vLLM engine metrics](https://docs.datarobot.com/en/docs/workload-api/create-workloads/deploy-llm-gpu.html.md#collect-vllm-engine-metrics). A running Workload can be viewed in Console at `https://app.datarobot.com/console-nextgen/workloads//overview`. Traces appear on the Monitoring > Data exploration tab. For more information, see the [Data exploration](https://docs.datarobot.com/en/docs/workload-api/monitor-workloads/monitoring-ui/data-exploration.html.md) documentation. ## Collect vLLM engine metrics This section is optional. Logs, service statistics, request traces, and resource metrics (including GPU utilization) are already collected without it. Add it only for vLLM's engine-internal metrics, such as KV-cache utilization and queue depth, on the Monitoring > OTel metrics surface. vLLM exposes those metrics only on a Prometheus `/metrics` endpoint (pull-based), while the platform's OTel metrics surface ingests OTLP (push). To collect them, add a bridge as a sidecar: a second, non-primary container that scrapes vLLM's `/metrics` over `localhost` and re-exports the values over OTLP using the OpenTelemetry SDK (see [Instrument a Workload with OpenTelemetry](https://docs.datarobot.com/en/docs/workload-api/monitor-workloads/instrument-with-otel.html.md#instrument-metrics)). Push the bridge image to a registry the platform can pull from, as described in [Image URI validation](https://docs.datarobot.com/en/docs/workload-api/build-artifacts/artifacts-concepts.html.md#image-uri-validation). ``` # Sidecar: scrape vLLM's Prometheus /metrics and re-export as OTLP metrics via the OTel SDK. - name: vllm-metrics-bridge imageUri: /vllm-metrics-bridge:latest environmentVars: - name: USER value: bridge - name: HOME value: /tmp # OTEL_EXPORTER_OTLP_ENDPOINT and its auth headers are injected by the platform. ``` Give every container in the group a `resourceAllocation` entry, dividing the bundle's capacity between them and keeping the totals within the bundle's CPU and RAM.`memory` accepts 1000-based units ( `B`, `KB`, `MB`, `GB`) or a raw byte integer; Kubernetes-style binary suffixes ( `Mi`, `Gi`) return a validation error.`gpu` sets GPU count only: ``` runtime: containerGroups: - name: default replicaCount: 1 resourceBundles: - gpu.medium containers: - name: vllm-server resourceAllocation: {cpu: 3, memory: "22GB", gpu: 1} - name: vllm-metrics-bridge resourceAllocation: {cpu: 0.5, memory: "2GB"} ``` ## Scale the Workload Update replicas or autoscaling via `PATCH /workloads/{workload_id}/settings`. See [Runtime settings](https://docs.datarobot.com/en/docs/workload-api/operate-workloads/runtime-settings.html.md) for the field reference and scaling metrics. Two considerations apply to a model of this size. Keep `minReplicaCount` at `1` or higher, since `HF_HOME` points at the container's own ephemeral filesystem and a replica scaled from zero repeats both the image pull and the full weight download before it can serve traffic. Organization-level caps also apply to concurrent Workloads and replicas: a `PATCH` that would exceed either cap returns `403`, so check current usage before scaling aggressively.