Batch agent evaluation¶

When an App Framework project includes the evaluation component, the experimentation dashboard can run batch evaluations against the agent. Each run sends test prompts to the agent OpenAI-compatible endpoint and scores every response using the NeMo Evaluator BYOB (Bring Your Own Benchmark) framework. The evaluator treats the agent as a black box—only the final OpenAI-compatible response is scored. Each run uses exactly one benchmark.

For installation, GUI access, and port configuration, see the DataRobot experimentation plugin landing page.

Enable batch evaluation workflows¶

Batch evaluation workflows in the GUI are controlled by the --enable-evaluation flag. Without it, the dashboard exposes Traces only—even when the evaluation component is installed in the project.

Pass the flag when starting the dashboard:

dr xp --enable-evaluation --entity-id <USE_CASE_ID> 

Alternatively, set the environment variable before launch:

export DR_EXPERIMENT_ENABLE_EVALUATION=true
dr xp --entity-id <USE_CASE_ID> 

For Agentic Starter projects that use dr run dev or task infra:dev, add the variable to the project root .env file instead. Taskfiles load .env automatically:

# .env (project root)
DR_EXPERIMENT_ENABLE_EVALUATION=true 

Bundled startup runs dr xp --plain-output without --enable-evaluation. The .env variable enables the Evaluation tab without changing the Taskfile.

When the evaluation component is present and evaluation is enabled, /api/info reports "features":["traces","evaluation"], and the Evaluation tab appears in the GUI. When disabled, features contains "traces" only, and evaluation API routes return 404.

If a dashboard is already running without the flag, stop it and restart with --enable-evaluation. See Evaluation troubleshooting for common symptoms.

前提条件¶

Judge-free benchmarks (answer_correctness, instruction_following, prompt_injection, pii_leakage, tool_grounding) run without judge credentials.

Run an evaluation¶

1. Install af-component-base, then add the evaluation component to the project (one-time):

   dr component add https://github.com/datarobot-community/af-component-evaluation . 
   

2. From the component directory (for example evaluations/), run task install.

3. Start the dashboard with batch evaluation workflows enabled:

   dr xp --enable-evaluation --entity-id <USE_CASE_ID> 
   

4. From the dr xp dashboard, select a pipeline, dataset, and agent endpoint, then start the run.

5. Alternatively, from the project root:

   dr task run evaluations:eval -- \
     --endpoint http://localhost:8842/v1 \
     --pipeline answer_quality.yaml \
     --dataset user_datasets/sample_answer_quality.json 
   

   Replace evaluations with the component task namespace if the folder was renamed during setup. Run dr task compose && task --list if the namespace is unknown.

Benchmarks at a glance¶

Each run uses one pipeline YAML from the component user_pipelines/ directory. Three benchmarks use an LLM judge; five use deterministic checks and need no judge model.

Key concepts¶

• Judge-based vs judge-free. Judge-based benchmarks call an LLM to grade responses; judge-free benchmarks use deterministic checks and do not require a judge model. Given the same agent response, a deterministic check produces the same score.
• Pass threshold. A case passes at score >= 0.5. Cases that cannot be scored (for example, when a judge call fails) are marked inconclusive and excluded from pass rates.
• Output vs internals. This workflow batch-tests agent outputs as a black box. To inspect tool calls, trajectories, or RAG retrieval, use NAT /evaluate instead. See NAT vs. NeMo.
• Pre-release vs runtime. Batch evaluation catches regressions on a fixed dataset before release. Runtime guardrails enforce policy on live traffic; they solve different problems and are not substitutes.

Evaluation troubleshooting¶

Further reading¶

See also¶

• DataRobot experimentation plugin—install, configure, and open the GUI.
• Local tracing—inspect OpenTelemetry spans during development.
• af-component-evaluation repository—evaluation component source and reference docs.